Oct 30 02:40:12.483171 ip-10-0-136-238 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:40:12.870805 ip-10-0-136-238 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:12.870805 ip-10-0-136-238 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:40:12.870805 ip-10-0-136-238 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:12.870805 ip-10-0-136-238 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:12.876265 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876184    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:12.876265 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876264    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:12.876265 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876268    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876272    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876275    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876278    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876280    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876283    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876286    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876288    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876291    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876294    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876296    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876300    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876308    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876311    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876313    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876317    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876322    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876325    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876328    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:12.876361 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876330    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876333    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876336    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876339    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876341    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876344    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876347    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876349    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876352    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876354    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876357    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876359    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876362    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876365    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876368    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876371    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876374    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876376    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876379    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876381    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:12.876796 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876384    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876386    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876389    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876391    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876394    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876397    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876399    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876402    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876406    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876409    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876413    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876415    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876418    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876421    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876423    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876426    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876429    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876432    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876434    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:12.877275 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876437    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876440    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876443    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876445    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876448    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876451    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876454    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876456    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876459    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876461    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876464    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876466    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876469    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876471    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876474    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876476    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876479    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876481    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876484    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876486    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:12.877720 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876489    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876491    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876494    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876497    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876499    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876502    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876505    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876508    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876512    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876515    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876518    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876521    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876523    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876526    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876529    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876531    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876534    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876536    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876539    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:12.878186 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876542    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876544    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876547    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876549    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876552    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876555    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.876558    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877329    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877336    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877339    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877341    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877344    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877347    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877349    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877352    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877355    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877359    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877362    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877365    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:12.878625 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877369    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877372    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877374    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877377    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877380    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877383    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877385    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877388    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877390    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877393    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877396    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877398    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877401    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877404    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877406    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877410    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877414    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877417    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877420    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:12.879087 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877422    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877425    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877428    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877431    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877433    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877436    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877438    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877441    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877443    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877448    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877452    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877455    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877458    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877461    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877465    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877468    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877470    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877473    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877475    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:12.879535 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877478    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877481    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877483    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877486    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877489    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877492    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877494    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877497    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877503    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877506    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877509    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877512    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877514    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877517    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877519    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877522    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877524    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877527    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877529    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:12.880038 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877532    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877535    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877537    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877540    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877543    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877546    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877548    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877551    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877554    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877557    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877559    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877562    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877564    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877567    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877569    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877572    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877575    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877577    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877580    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877582    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:12.880563 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877585    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877588    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877591    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877593    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877595    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877598    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877600    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877603    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877605    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877608    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877610    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877613    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877615    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877618    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877622    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877624    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.877627    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878105    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878110    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878114    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:12.881085 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878117    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878120    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878123    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878126    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878129    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878131    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878134    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878137    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878139    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878142    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878144    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878147    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878149    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878152    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878155    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878158    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878160    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878163    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878165    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878168    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:12.881548 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878171    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878173    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878176    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878178    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878183    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878186    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878189    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878192    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878194    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878197    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878200    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878202    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878205    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878208    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878210    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878213    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878216    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878218    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878221    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878223    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:12.882055 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878226    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878228    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878231    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878233    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878235    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878238    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878241    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878244    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878246    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878249    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878251    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878254    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878256    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878259    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878261    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878263    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878267    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878270    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878272    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:12.882576 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878275    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878278    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878281    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878284    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878286    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878289    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878292    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878294    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878297    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878300    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878302    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878305    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878308    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878311    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878313    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878316    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878318    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878321    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878323    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:12.883035 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878326    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878328    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878331    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878333    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878336    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878338    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878341    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878343    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878346    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878348    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878352    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878355    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878358    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878360    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878363    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878365    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878368    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878374    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878376    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878381    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:12.883472 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878385    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878388    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878390    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878393    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.878395    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879122    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879131    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879138    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879143    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879148    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879151    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879155    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879159    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879163    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879166    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879170    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879174    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879177    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879180    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879183    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879186    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879189    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879192    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:40:12.883971 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879196    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879199    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879202    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879205    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879208    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879212    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879217    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879221    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879224    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879227    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879230    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879233    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879236    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879239    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879244    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879247    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879250    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879253    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879256    2576 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879259    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879264    2576 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879267    2576 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879270    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879273    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:40:12.884496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879276    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879281    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879284    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879287    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879290    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879294    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879296    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879300    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879303    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879306    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879309    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879312    2576 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879316    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879319    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879323    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879327    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879330    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879333    2576 flags.go:64] FLAG: --help="false"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879336    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-136-238.ec2.internal"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879339    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879342    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879345    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879348    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:40:12.885086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879351    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879354    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879357    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879360    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879363    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879366    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879369    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879372    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879375    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879378    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879381    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879385    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879388    2576 flags.go:64] FLAG: --lock-file=""
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879391    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879394    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879397    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879403    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879406    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879409    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879412    2576 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879415    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879418    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879421    2576 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879425    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879429    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:40:12.885622 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879432    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879436    2576 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879439    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879442    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879445    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879448    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879451    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879454    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879457    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879464    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879467    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879470    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879473    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879476    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879479    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879482    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879485    2576 flags.go:64] FLAG: --port="10250"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879488    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879491    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ddbafb71124ee23e"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879495    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879498    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879501    2576 flags.go:64] FLAG: --register-node="true"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879504    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879508    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879511    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879514    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:40:12.886249 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879517    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879521    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879524    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879527    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879531    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879534    2576 flags.go:64] FLAG: --runonce="false"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879537    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879540    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879560    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879564    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879568    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879571    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879575    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879578    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879581    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879584    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879587    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879589    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879592    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879595    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879599    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879604    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879607    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879610    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879613    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:40:12.887158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879625    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879630    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879633    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879637    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879640    2576 flags.go:64] FLAG: --v="2"
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879645    2576 flags.go:64] FLAG: --version="false"
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879649    2576 flags.go:64] FLAG: --vmodule=""
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879653    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.879656    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879787    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879791    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879796    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879799    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879802    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879804    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879807    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879809    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879812    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879814    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879817    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879820    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:12.887893 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879823    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879826    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879829    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879831    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879834    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879837    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879840    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879842    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879858    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879862    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879867    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879871    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879875    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879879    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879883    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879886    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879888    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879892    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879896    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:12.888397 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879900    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879903    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879907    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879910    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879913    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879916    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879918    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879921    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879924    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879926    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879929    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879931    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879934    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879937    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879939    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879942    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879944    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879947    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879950    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879952    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:12.888841 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879955    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879957    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879960    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879962    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879965    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879967    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879970    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879974    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879977    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879981    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879983    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879986    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879988    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879991    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879993    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.879998    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880001    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880004    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880006    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:12.889325 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880009    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880012    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880014    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880017    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880020    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880022    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880025    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880027    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880030    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880033    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880035    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880039    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880043    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880046    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880049    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880052    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880055    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880058    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880060    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:12.889768 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880063    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880067    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880070    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880073    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880075    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880078    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880081    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880083    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880086    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880089    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880092    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880095    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880097    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880100    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880102    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880105    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:12.890251 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880107    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.880134    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880170    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880174    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880177    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880180    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880183    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880185    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880188    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880191    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880193    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880196    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880199    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880201    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880204    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880206    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:12.890627 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880209    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880213    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880215    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880218    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880221    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880224    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880227    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880229    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880232    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880234    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880238    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880240    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880243    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880246    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880248    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880251    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880253    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880256    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880258    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880261    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:12.891022 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880264    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880266    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880270    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880272    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880275    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880277    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880280    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880283    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880286    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880288    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880291    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880293    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880296    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880299    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880302    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880305    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880308    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880310    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880314    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:12.891481 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880318    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880321    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880324    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880328    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880331    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880334    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880336    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880339    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880341    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880344    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880346    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880349    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880352    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880354    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880356    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880359    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880362    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880364    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880367    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880369    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:12.892064 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880371    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880374    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880377    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880379    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880382    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880384    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880388    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880390    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880393    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880395    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880399    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880401    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880404    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880406    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880409    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880413    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880415    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880418    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880421    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:12.892560 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880423    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880426    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880428    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880431    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880434    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880437    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880440    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880442    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880445    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880447    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880450    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880453    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880457    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.880461    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.888234    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.888248    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.888871    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.888877    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888937    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:12.893032 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888944    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888947    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888951    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888954    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888957    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888960    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888963    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888966    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888969    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888972    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888974    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888977    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888980    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888982    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888985    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888988    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888990    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888993    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888995    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:12.893488 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.888998    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889001    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889003    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889006    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889009    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889013    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889016    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889019    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889022    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889025    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889028    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889031    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889034    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889037    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889039    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889042    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889044    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889047    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889049    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:12.893929 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889052    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889054    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889057    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889060    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889063    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889065    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889068    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889070    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889073    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889076    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889079    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889081    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889084    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889087    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889089    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889092    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889094    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889097    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889099    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889102    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:12.894362 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889104    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889107    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889110    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889112    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889115    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889118    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889120    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889123    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889126    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889130    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889134    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889136    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889139    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889142    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889145    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889148    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889150    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889153    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889156    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889158    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:12.894812 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889162    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889164    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889167    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889170    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889172    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889175    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889177    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889180    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889182    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889185    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889187    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889190    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889193    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889195    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889198    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889200    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889203    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889205    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889208    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889211    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:12.895307 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889213    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889216    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889218    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889221    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889223    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889226    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889228    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.889255    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889386    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889392    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889394    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889398    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889401    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889403    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889406    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:12.895779 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889409    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889411    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889414    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889417    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889419    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889422    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889424    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889427    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889430    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889432    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889435    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889438    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889440    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889443    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889446    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889448    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889451    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889453    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889456    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889458    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:12.896205 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889461    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889463    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889466    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889469    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889471    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889474    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889477    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889479    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889482    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889484    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889487    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889490    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889492    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889495    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889497    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889500    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889503    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889505    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889508    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889510    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:12.896730 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889513    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889515    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889519    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889523    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889526    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889529    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889531    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889534    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889537    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889539    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889542    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889545    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889547    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889550    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889552    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889554    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889557    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889561    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889564    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:12.897212 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889567    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889569    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889571    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889574    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889576    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889580    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889583    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889585    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889588    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889591    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889593    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889595    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889600    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889603    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889605    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889608    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889610    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889613    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889615    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889618    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:12.897642 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889621    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889623    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889627    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889630    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889633    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889636    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889639    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889641    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889644    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889646    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889651    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889653    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889656    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889659    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889661    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889664    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889667    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889669    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889672    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:12.898143 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889675    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.889698    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889727    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889731    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889734    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889737    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889740    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889742    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889746    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889748    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889751    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889754    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889756    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889759    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889761    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889764    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:12.898590 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889767    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889769    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889772    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889774    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889777    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889780    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889782    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889785    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889789    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889791    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889794    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889798    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889802    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889805    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889807    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889810    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889814    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889816    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889819    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:12.898964 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889822    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889825    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889828    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889831    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889833    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889836    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889839    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889841    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889859    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889862    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889865    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889868    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889871    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889873    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889876    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889879    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889881    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889884    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889887    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:12.899402 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889889    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889892    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889896    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889898    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889901    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889904    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889906    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889909    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889912    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889915    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889917    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889921    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889924    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889929    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889932    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889936    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889939    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889942    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889944    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:12.899884 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889947    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889950    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889953    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889955    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889958    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889960    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889963    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889966    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889968    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889971    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889974    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889976    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889979    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889981    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889984    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889989    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889992    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889995    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889997    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:12.900357 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.889999    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890002    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890005    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890008    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890010    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890013    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890016    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890019    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890021    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890025    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890029    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890032    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890034    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890037    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890039    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:12.890042    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.890088    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.892297    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.893226    2576 server.go:1014] "Starting client certificate rotation"
Oct 30 02:40:12.900792 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.893329    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:40:12.901325 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.893361    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:40:12.901325 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.895240    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:12.901325 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.896324    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:12.906247 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.906233    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:40:12.909621 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.909603    2576 log.go:25] "Validated CRI v1 image API"
Oct 30 02:40:12.910402 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.910378    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:40:12.913476 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.913458    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 abd2dc97-6d78-40fa-b413-ff827ec7bbcb:/dev/nvme0n1p3 b03fa22c-7d29-46f6-b93e-aa3ed126fed2:/dev/nvme0n1p4]
Oct 30 02:40:12.913526 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.913477    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 02:40:12.919963 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.919830    2576 manager.go:217] Machine: {Timestamp:2026-10-30 02:40:12.918535181 +0000 UTC m=+0.319106744 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105721 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2785ac5795e731f83b8784b9ee237b SystemUUID:ec2785ac-5795-e731-f83b-8784b9ee237b BootID:4fabfbc6-19a9-403c-96b3-1a00cf354f57 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d9:e9:7e:f7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d9:e9:7e:f7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:5c:1c:7a:af:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:40:12.919963 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.919961    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:40:12.920077 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.920034    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:40:12.921153 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.921131    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:40:12.922393 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.921164    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-238.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:40:12.923185 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.923165    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:40:12.923380 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.923227    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:40:12.923416 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.923409    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:12.923443 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.923428    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:12.924666 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.924654    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:40:12.928958 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.928946    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:40:12.929731 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.929721    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:40:12.930418 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.930408    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:40:12.930457 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.930423    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:40:12.930457 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.930432    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:40:12.930457 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.930441    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:40:12.930457 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.930450    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:40:12.930786 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.930775    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:12.933083 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.933064    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:40:12.940204 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.940187    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:12.941124 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.941109    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:40:12.941167 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.941141    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:40:12.941316 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.941306    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:40:12.941345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.941322    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:40:12.941345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.941329    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:40:12.941345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.941334    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:40:12.941345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.941340    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:40:12.941345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.941346    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:40:12.942005 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.941995    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:40:12.942036 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.942006    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:40:12.942036 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.942016    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:40:12.942036 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.942021    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:40:12.942036 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.942031    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:40:12.942159 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.942040    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:40:12.943330 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.943321    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:40:12.943366 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.943332    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:40:12.948994 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.948976    2576 server.go:1263] "Started kubelet"
Oct 30 02:40:12.949178 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.949114    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:40:12.949320 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.949240    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:40:12.949396 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.949367    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:40:12.950654 ip-10-0-136-238 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:40:12.951477 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.951204    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:40:12.953826 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.953809    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:40:12.956279 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:12.956230    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 02:40:12.961992 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.961966    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:40:12.962651 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.962635    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:40:12.962802 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:12.962768    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:12.963580 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.963535    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:40:12.963580 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.963542    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:40:12.963580 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.963574    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:40:12.963808 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.963695    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:40:12.963808 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.963703    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:40:12.964388 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.964369    2576 factory.go:55] Registering systemd factory
Oct 30 02:40:12.964482 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.964393    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:40:12.964699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.964683    2576 factory.go:153] Registering CRI-O factory
Oct 30 02:40:12.964734 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.964704    2576 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:40:12.964785 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.964773    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:40:12.964816 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.964801    2576 factory.go:103] Registering Raw factory
Oct 30 02:40:12.964870 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.964820    2576 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:40:12.965941 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.965925    2576 manager.go:319] Starting recovery of all containers
Oct 30 02:40:12.977415 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.977390    2576 manager.go:324] Recovery completed
Oct 30 02:40:12.977526 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.977503    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-238.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 02:40:12.978899 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:12.977502    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-238.ec2.internal.18e32e538993f772  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-238.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-238.ec2.internal,},FirstTimestamp:2026-10-30 02:40:12.94892837 +0000 UTC m=+0.349499934,LastTimestamp:2026-10-30 02:40:12.94892837 +0000 UTC m=+0.349499934,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-238.ec2.internal,}"
Oct 30 02:40:12.979154 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:12.979123    2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 02:40:12.982250 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.982236    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:12.984039 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.984013    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:12.984116 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.984047    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:12.984116 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.984057    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:12.985156 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.985135    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:40:12.985156 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.985147    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:40:12.985247 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.985167    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:40:12.985247 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.985139    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:12.986106 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.986086    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:40:12.986187 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.986086    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:12.986187 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.986132    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:12.986187 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.986142    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:12.987470 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.987455    2576 policy_none.go:50] "Start"
Oct 30 02:40:12.988124 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.988107    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:40:12.988171 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.988144    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:40:12.990302 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:12.990289    2576 policy_none.go:44] "Start"
Oct 30 02:40:12.991112 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:12.991078    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-238.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 02:40:13.004672 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.004651    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cbz44"
Oct 30 02:40:13.011081 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.011063    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cbz44"
Oct 30 02:40:13.023249 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.023220    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 02:40:13.023344 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.023245    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 02:40:13.023344 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.023202    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-238.ec2.internal.18e32e538babb0db  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-238.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-238.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-238.ec2.internal,},FirstTimestamp:2026-10-30 02:40:12.984037595 +0000 UTC m=+0.384609158,LastTimestamp:2026-10-30 02:40:12.984037595 +0000 UTC m=+0.384609158,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-238.ec2.internal,}"
Oct 30 02:40:13.023344 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.023245    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-238.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 02:40:13.026424 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.026397    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:40:13.032496 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.032479    2576 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:40:13.032574 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.032517    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:40:13.032574 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.032528    2576 server.go:87] "Starting device plugin registration server"
Oct 30 02:40:13.032782 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.032770    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:40:13.032839 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.032782    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:40:13.032931 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.032901    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:40:13.032992 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.032974    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:40:13.032992 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.032985    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:40:13.033578 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.033548    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:40:13.033648 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.033605    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:13.108086 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.107917    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:40:13.109557 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.109398    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:40:13.109557 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.109420    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:40:13.109557 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.109445    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:40:13.109557 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.109479    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 02:40:13.125325 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.125277    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:13.133964 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.133948    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:13.134735 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.134719    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:13.134817 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.134750    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:13.134817 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.134765    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:13.134817 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.134792    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.154510 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.154493    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.154585 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.154521    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-238.ec2.internal\": node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:13.194986 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.194966    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:13.210520 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.210491    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"]
Oct 30 02:40:13.210611 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.210550    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:13.211454 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.211439    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:13.211524 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.211469    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:13.211524 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.211479    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:13.213181 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.213169    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:13.213329 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.213315    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.213389 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.213353    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:13.213941 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.213926    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:13.213998 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.213953    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:13.213998 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.213967    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:13.213998 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.213974    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:13.213998 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.213992    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:13.214143 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.214006    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:13.215828 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.215811    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.215902 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.215858    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:13.216563 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.216546    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:13.216642 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.216576    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:13.216642 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.216591    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:13.239693 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.239669    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-238.ec2.internal\" not found" node="ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.242654 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.242633    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-238.ec2.internal\" not found" node="ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.264875 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.264833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.265012 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.264879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.265012 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.264951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f18f0dafccedcaa5cd8761c40142410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-238.ec2.internal\" (UID: \"4f18f0dafccedcaa5cd8761c40142410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.295350 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.295319    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:13.365574 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.365542    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f18f0dafccedcaa5cd8761c40142410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-238.ec2.internal\" (UID: \"4f18f0dafccedcaa5cd8761c40142410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.365574 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.365575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.365731 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.365595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.365731 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.365666    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.365731 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.365695    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f18f0dafccedcaa5cd8761c40142410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-238.ec2.internal\" (UID: \"4f18f0dafccedcaa5cd8761c40142410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.365731 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.365666    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.395628 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.395558    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:13.496312 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.496281    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:13.544464 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.544445    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.547584 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.547569    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:13.597258 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.597233    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:13.697606 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.697576    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:13.799817 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.799785    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:13.893172 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.893131    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 02:40:13.893657 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:13.893277    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:40:13.900688 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:13.900670    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:14.001943 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:14.001863    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:14.017621 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.017586    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:35:13 +0000 UTC" deadline="2028-04-23 22:42:11.921812633 +0000 UTC"
Oct 30 02:40:14.017621 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.017616    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13004h1m57.904200179s"
Oct 30 02:40:14.055744 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.055706    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:14.057466 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.057442    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:14.057577 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.057487    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:14.057577 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.057507    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:14.102298 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:14.102261    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:14.152761 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:14.152724    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6521a54728e46ede7b6de87a0d41821f.slice/crio-e012fcef3d9f5dc2ef3cb13ffffb262b36d02196281a59fde68bfb10f6f4d84c WatchSource:0}: Error finding container e012fcef3d9f5dc2ef3cb13ffffb262b36d02196281a59fde68bfb10f6f4d84c: Status 404 returned error can't find the container with id e012fcef3d9f5dc2ef3cb13ffffb262b36d02196281a59fde68bfb10f6f4d84c
Oct 30 02:40:14.154695 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:14.154670    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f18f0dafccedcaa5cd8761c40142410.slice/crio-334191dd0df23d9916c528378d51f78f533f6ca3987f33bec6600101f20a4bd5 WatchSource:0}: Error finding container 334191dd0df23d9916c528378d51f78f533f6ca3987f33bec6600101f20a4bd5: Status 404 returned error can't find the container with id 334191dd0df23d9916c528378d51f78f533f6ca3987f33bec6600101f20a4bd5
Oct 30 02:40:14.160362 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.160342    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:40:14.203117 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:14.203094    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:14.221416 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.221397    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:14.305685 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:14.305633    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:14.405956 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:14.405915    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:14.471963 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.471940    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:14.564045 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.563980    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:14.590294 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.590265    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:14.592171 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.592150    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:14.613673 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.613654    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:14.643213 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.643190    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:14.788016 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.787985    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:14.932158 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.932090    2576 apiserver.go:52] "Watching apiserver"
Oct 30 02:40:14.962735 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.962706    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 02:40:14.964929 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.964904    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:40:14.965296 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.965275    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-vpwcs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5","openshift-image-registry/node-ca-hqwcj","kube-system/konnectivity-agent-5l2pw","openshift-multus/multus-dmfvd","openshift-network-diagnostics/network-check-target-tb8xt","kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal","openshift-cluster-node-tuning-operator/tuned-flkp8","openshift-dns/node-resolver-qvbmn","openshift-multus/multus-additional-cni-plugins-8pbzl","kube-system/global-pull-secret-syncer-hzqcm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal","openshift-multus/network-metrics-daemon-pt72w","openshift-network-operator/iptables-alerter-4lscq"]
Oct 30 02:40:14.968119 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.968089    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:14.970535 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.970509    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:14.970648 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:14.970593    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:14.972951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.972934    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:14.975386 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.975365    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:14.978023 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.977977    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:14.980296 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.980274    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:14.980397 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:14.980365    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:14.982818 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.982796    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:14.986196 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.986137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:14.987542 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.987300    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:40:14.987542 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.987316    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:40:14.987542 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.987457    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:40:14.987542 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.987501    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:40:14.987790 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.987577    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5k7wg\""
Oct 30 02:40:14.987790 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.987598    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:40:14.990215 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.990189    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:40:14.990538 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.990520    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:14.993670 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.993647    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:14.998131 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:14.998036    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:14.998131 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:14.998097    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:15.000691 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.000561    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.012094 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.012075    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:40:15.012201 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.012140    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x44h8"
Oct 30 02:40:15.065472 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.065450    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:40:15.075878 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.075841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-cni-binary-copy\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.075988 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.075892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-k8s-cni-cncf-io\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.075988 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.075911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-multus-certs\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.075988 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.075934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-ovn\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.076133 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.075987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-node-log\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.076133 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-bin\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.076133 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-config\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.076133 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-run\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.076133 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-netns\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.076133 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-kubelet\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.076347 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-systemd-units\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.076347 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076171    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-var-lib-kubelet\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.076347 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-host\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.076347 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfg7\" (UniqueName: \"kubernetes.io/projected/fa5e5905-3e48-434f-8153-fd40123a614a-kube-api-access-fvfg7\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.076347 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076227    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caeb5235-93eb-4b90-8372-1c59c807485f-agent-certs\") pod \"konnectivity-agent-5l2pw\" (UID: \"caeb5235-93eb-4b90-8372-1c59c807485f\") " pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:15.076347 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:15.076347 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-bin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.076568 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzrsj\" (UniqueName: \"kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj\") pod \"network-check-target-tb8xt\" (UID: \"00bc8689-1f9b-40a7-9f19-972395d1bc69\") " pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:15.076568 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6996\" (UniqueName: \"kubernetes.io/projected/960da804-13fa-4a2b-8837-1dd6f8ca83c3-kube-api-access-t6996\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:15.076568 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-registration-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.076568 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-kubernetes\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.076568 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.076568 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-slash\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.076568 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysconfig\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.076568 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076536    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.076814 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-hostroot\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.076814 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076610    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/960da804-13fa-4a2b-8837-1dd6f8ca83c3-host-slash\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:15.076814 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.076814 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-env-overrides\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.076814 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-modprobe-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.076814 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.076814 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjrl\" (UniqueName: \"kubernetes.io/projected/af778722-0e75-4ddd-9819-d783cfa858e1-kube-api-access-jjjrl\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:15.077064 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076828    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-multus\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.077064 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-hosts-file\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:15.077064 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27s6x\" (UniqueName: \"kubernetes.io/projected/cb4e2245-df10-43f4-a52a-433997bc0ca4-kube-api-access-27s6x\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.077064 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-kubelet\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.077064 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.077064 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.076986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-script-lib\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.077064 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077006    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbjc\" (UniqueName: \"kubernetes.io/projected/816aaac5-8df6-402e-9367-a2ea2f8ca686-kube-api-access-6sbjc\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.077064 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-socket-dir-parent\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.077064 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-etc-kubernetes\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-socket-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-netns\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-etc-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-log-socket\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-conf\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077219    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnvvc\" (UniqueName: \"kubernetes.io/projected/1ec92445-aa69-4865-9d4e-8a1423bf7073-kube-api-access-wnvvc\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qqnb\" (UniqueName: \"kubernetes.io/projected/2c04bf2f-3e60-442a-8639-10feae0a3f96-kube-api-access-6qqnb\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af778722-0e75-4ddd-9819-d783cfa858e1-host\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af778722-0e75-4ddd-9819-d783cfa858e1-serviceca\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caeb5235-93eb-4b90-8372-1c59c807485f-konnectivity-ca\") pod \"konnectivity-agent-5l2pw\" (UID: \"caeb5235-93eb-4b90-8372-1c59c807485f\") " pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/960da804-13fa-4a2b-8837-1dd6f8ca83c3-iptables-alerter-script\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:15.077456 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077377    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovn-node-metrics-cert\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077479    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-kubelet-config\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-conf-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvbf\" (UniqueName: \"kubernetes.io/projected/92515440-cc1f-4e4d-b45e-6afaff621108-kube-api-access-qnvbf\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077559    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-sys-fs\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077608    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-netd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-dbus\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-lib-modules\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-tuned\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-cnibin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvz9z\" (UniqueName: \"kubernetes.io/projected/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-kube-api-access-zvz9z\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077711    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-tmp\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077736    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-cnibin\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.077951 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-system-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-multus-daemon-config\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-systemd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-var-lib-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-systemd\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-sys\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.077997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.078025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-tmp-dir\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.078044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.078068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-device-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.078089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.078699 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.078119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:15.093980 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.093961    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:40:15.094079 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.094060    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:15.094626 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.094516    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:40:15.094626 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.094562    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:40:15.094626 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.094595    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:40:15.094626 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.094625    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gf5d6\""
Oct 30 02:40:15.094896 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.094757    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:40:15.094896 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.094762    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:40:15.095202 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095181    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j5lkg\""
Oct 30 02:40:15.095202 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095181    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:40:15.095345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095188    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:15.095345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095203    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:40:15.095345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095247    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:40:15.095345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095256    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4pml\""
Oct 30 02:40:15.095345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095192    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:40:15.095345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095287    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:40:15.095345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095325    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:40:15.095345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095326    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:15.095345 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095345    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:40:15.095751 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095352    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r7gc2\""
Oct 30 02:40:15.095751 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095292    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-prmc8\""
Oct 30 02:40:15.095751 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095388    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-86l9w\""
Oct 30 02:40:15.095751 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095402    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:40:15.095751 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095407    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ph87b\""
Oct 30 02:40:15.095751 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095408    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:15.095751 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095430    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:40:15.095751 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095456    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:40:15.096179 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095919    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:40:15.096179 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.095946    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hhdzd\""
Oct 30 02:40:15.115024 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.114978    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal" event={"ID":"4f18f0dafccedcaa5cd8761c40142410","Type":"ContainerStarted","Data":"334191dd0df23d9916c528378d51f78f533f6ca3987f33bec6600101f20a4bd5"}
Oct 30 02:40:15.116491 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.116458    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal" event={"ID":"6521a54728e46ede7b6de87a0d41821f","Type":"ContainerStarted","Data":"e012fcef3d9f5dc2ef3cb13ffffb262b36d02196281a59fde68bfb10f6f4d84c"}
Oct 30 02:40:15.178565 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-log-socket\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.178716 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-conf\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.178716 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnvvc\" (UniqueName: \"kubernetes.io/projected/1ec92445-aa69-4865-9d4e-8a1423bf7073-kube-api-access-wnvvc\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.178716 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qqnb\" (UniqueName: \"kubernetes.io/projected/2c04bf2f-3e60-442a-8639-10feae0a3f96-kube-api-access-6qqnb\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:15.178716 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af778722-0e75-4ddd-9819-d783cfa858e1-host\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:15.178716 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af778722-0e75-4ddd-9819-d783cfa858e1-serviceca\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:15.178716 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caeb5235-93eb-4b90-8372-1c59c807485f-konnectivity-ca\") pod \"konnectivity-agent-5l2pw\" (UID: \"caeb5235-93eb-4b90-8372-1c59c807485f\") " pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:15.178716 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/960da804-13fa-4a2b-8837-1dd6f8ca83c3-iptables-alerter-script\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovn-node-metrics-cert\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-kubelet-config\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-conf-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvbf\" (UniqueName: \"kubernetes.io/projected/92515440-cc1f-4e4d-b45e-6afaff621108-kube-api-access-qnvbf\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-sys-fs\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178925    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-netd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.178974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-dbus\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-lib-modules\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-tuned\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-cnibin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvz9z\" (UniqueName: \"kubernetes.io/projected/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-kube-api-access-zvz9z\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179097    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-tmp\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.179123 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-cnibin\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179175    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-system-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-multus-daemon-config\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-systemd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-var-lib-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-systemd\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-sys\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-tmp-dir\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-device-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-cni-binary-copy\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-k8s-cni-cncf-io\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179548    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-multus-certs\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.179983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-ovn\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-node-log\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-bin\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-config\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-run\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-netns\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-kubelet\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-systemd-units\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-var-lib-kubelet\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-host\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfg7\" (UniqueName: \"kubernetes.io/projected/fa5e5905-3e48-434f-8153-fd40123a614a-kube-api-access-fvfg7\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caeb5235-93eb-4b90-8372-1c59c807485f-agent-certs\") pod \"konnectivity-agent-5l2pw\" (UID: \"caeb5235-93eb-4b90-8372-1c59c807485f\") " pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179896    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-bin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.179975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrsj\" (UniqueName: \"kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj\") pod \"network-check-target-tb8xt\" (UID: \"00bc8689-1f9b-40a7-9f19-972395d1bc69\") " pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6996\" (UniqueName: \"kubernetes.io/projected/960da804-13fa-4a2b-8837-1dd6f8ca83c3-kube-api-access-t6996\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180033    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-registration-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.180815 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-kubernetes\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-slash\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysconfig\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-hostroot\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/960da804-13fa-4a2b-8837-1dd6f8ca83c3-host-slash\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-env-overrides\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-modprobe-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjrl\" (UniqueName: \"kubernetes.io/projected/af778722-0e75-4ddd-9819-d783cfa858e1-kube-api-access-jjjrl\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-multus\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-hosts-file\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27s6x\" (UniqueName: \"kubernetes.io/projected/cb4e2245-df10-43f4-a52a-433997bc0ca4-kube-api-access-27s6x\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-kubelet\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180548    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.181615 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-script-lib\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180610    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbjc\" (UniqueName: \"kubernetes.io/projected/816aaac5-8df6-402e-9367-a2ea2f8ca686-kube-api-access-6sbjc\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-socket-dir-parent\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-etc-kubernetes\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-socket-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-netns\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-etc-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180866    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-etc-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.180941    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-log-socket\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.181086    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-conf\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.181597    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af778722-0e75-4ddd-9819-d783cfa858e1-host\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.182354    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-run\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.182440 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.182415    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-system-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.183212 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.182482    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-var-lib-kubelet\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.183212 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.182488    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-systemd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.183212 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.182518    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-kubelet-config\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:15.183212 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.182529    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-var-lib-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.183212 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.182839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af778722-0e75-4ddd-9819-d783cfa858e1-serviceca\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:15.183212 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.182894    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-systemd-units\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.183212 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.182420    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-netns\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.183212 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183010    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-conf-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.183212 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183066    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.183212 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183172    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-sys-fs\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.183212 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183217    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183228    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183260    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-k8s-cni-cncf-io\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183273    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-slash\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183303    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-socket-dir-parent\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183306    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysconfig\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183311    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-netd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183348    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-etc-kubernetes\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183392    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-device-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183399    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.183423    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183434    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-hostroot\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183457    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/960da804-13fa-4a2b-8837-1dd6f8ca83c3-host-slash\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-netns\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.183494    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs podName:2c04bf2f-3e60-442a-8639-10feae0a3f96 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:15.683474248 +0000 UTC m=+3.084045816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs") pod "network-metrics-daemon-pt72w" (UID: "2c04bf2f-3e60-442a-8639-10feae0a3f96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183521    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-dbus\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:15.183583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183567    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-socket-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.184192 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183613    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-lib-modules\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.184192 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183668    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-sys\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.184192 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183704    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-bin\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.184192 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183754    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.184192 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183795    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-bin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.184192 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183836    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-cnibin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.184192 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183882    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:40:15.184502 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.184459    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-systemd\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.184559 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.184525    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-host\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.185530 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185129    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.185530 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185187    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-cnibin\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.185530 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185214    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.185530 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185457    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-modprobe-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.185530 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185453    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.185530 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185455    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caeb5235-93eb-4b90-8372-1c59c807485f-konnectivity-ca\") pod \"konnectivity-agent-5l2pw\" (UID: \"caeb5235-93eb-4b90-8372-1c59c807485f\") " pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:15.185530 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185504    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-ovn\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.185973 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185537    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.185973 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185619    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-kubernetes\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.185973 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.183348    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.185973 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185748    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-kubelet\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.185973 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185785    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-tmp-dir\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:15.185973 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.185837    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:15.185973 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185866    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-hosts-file\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:15.185973 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185883    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-node-log\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.185973 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.185906    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-kubelet\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.185973 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.185939    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret podName:a2bf413f-45b3-450c-b797-4e674e8da8ec nodeName:}" failed. No retries permitted until 2026-10-30 02:40:15.685905064 +0000 UTC m=+3.086476619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret") pod "global-pull-secret-syncer-hzqcm" (UID: "a2bf413f-45b3-450c-b797-4e674e8da8ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:15.186481 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.186051    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-registration-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.186481 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.186060    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-multus\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.186481 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.186100    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-multus-certs\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.187225 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.187206    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-multus-daemon-config\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.187641 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.187620    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-cni-binary-copy\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.187948 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.187880    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/960da804-13fa-4a2b-8837-1dd6f8ca83c3-iptables-alerter-script\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:15.188809 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.188785    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-config\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.188809 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.188798    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-tmp\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.189619 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.189562    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-tuned\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.189619 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.189585    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-script-lib\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.189803 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.189636    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.189803 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.189779    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.189939 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.189811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-env-overrides\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.190937 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.190918    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caeb5235-93eb-4b90-8372-1c59c807485f-agent-certs\") pod \"konnectivity-agent-5l2pw\" (UID: \"caeb5235-93eb-4b90-8372-1c59c807485f\") " pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:15.197978 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.197958    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovn-node-metrics-cert\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.203817 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.203784    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnvvc\" (UniqueName: \"kubernetes.io/projected/1ec92445-aa69-4865-9d4e-8a1423bf7073-kube-api-access-wnvvc\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.212940 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.212905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvz9z\" (UniqueName: \"kubernetes.io/projected/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-kube-api-access-zvz9z\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:15.213037 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.212984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qqnb\" (UniqueName: \"kubernetes.io/projected/2c04bf2f-3e60-442a-8639-10feae0a3f96-kube-api-access-6qqnb\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:15.217489 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.217176    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:15.217489 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.217203    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:15.217489 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.217218    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wzrsj for pod openshift-network-diagnostics/network-check-target-tb8xt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:15.217489 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.217306    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj podName:00bc8689-1f9b-40a7-9f19-972395d1bc69 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:15.717286535 +0000 UTC m=+3.117858114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wzrsj" (UniqueName: "kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj") pod "network-check-target-tb8xt" (UID: "00bc8689-1f9b-40a7-9f19-972395d1bc69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:15.219010 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.218984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27s6x\" (UniqueName: \"kubernetes.io/projected/cb4e2245-df10-43f4-a52a-433997bc0ca4-kube-api-access-27s6x\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.221220 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.220792    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbjc\" (UniqueName: \"kubernetes.io/projected/816aaac5-8df6-402e-9367-a2ea2f8ca686-kube-api-access-6sbjc\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.221220 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.220993    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjrl\" (UniqueName: \"kubernetes.io/projected/af778722-0e75-4ddd-9819-d783cfa858e1-kube-api-access-jjjrl\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:15.221902 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.221883    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvbf\" (UniqueName: \"kubernetes.io/projected/92515440-cc1f-4e4d-b45e-6afaff621108-kube-api-access-qnvbf\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.222210 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.222184    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfg7\" (UniqueName: \"kubernetes.io/projected/fa5e5905-3e48-434f-8153-fd40123a614a-kube-api-access-fvfg7\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.223274 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.223249    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6996\" (UniqueName: \"kubernetes.io/projected/960da804-13fa-4a2b-8837-1dd6f8ca83c3-kube-api-access-t6996\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:15.297363 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.297328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:15.299433 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.299399    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:15.307432 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.307408    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:15.313394 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.313369    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:15.313981 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:15.313798    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5e5905_3e48_434f_8153_fd40123a614a.slice/crio-03cfc82835843a9b804eab876a0e38b76a839fa5e125ba261219a303d233214f WatchSource:0}: Error finding container 03cfc82835843a9b804eab876a0e38b76a839fa5e125ba261219a303d233214f: Status 404 returned error can't find the container with id 03cfc82835843a9b804eab876a0e38b76a839fa5e125ba261219a303d233214f
Oct 30 02:40:15.315025 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:15.314996    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf778722_0e75_4ddd_9819_d783cfa858e1.slice/crio-72213ea7d0fd40202430cd7d816217cc4b53a7a5d9dd3ddffc64d809d2d94e35 WatchSource:0}: Error finding container 72213ea7d0fd40202430cd7d816217cc4b53a7a5d9dd3ddffc64d809d2d94e35: Status 404 returned error can't find the container with id 72213ea7d0fd40202430cd7d816217cc4b53a7a5d9dd3ddffc64d809d2d94e35
Oct 30 02:40:15.321188 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:15.321131    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaeb5235_93eb_4b90_8372_1c59c807485f.slice/crio-b9c9384fa29d31d5a3ff68f85599ee9ed4e2155eaefaf775e6a6ee97377bc2a7 WatchSource:0}: Error finding container b9c9384fa29d31d5a3ff68f85599ee9ed4e2155eaefaf775e6a6ee97377bc2a7: Status 404 returned error can't find the container with id b9c9384fa29d31d5a3ff68f85599ee9ed4e2155eaefaf775e6a6ee97377bc2a7
Oct 30 02:40:15.322288 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.322267    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:15.325868 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:15.325832    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92515440_cc1f_4e4d_b45e_6afaff621108.slice/crio-ee041b160d660ef171b3245c331d118a56c65e2460759aeb7cbe3a92a6080b79 WatchSource:0}: Error finding container ee041b160d660ef171b3245c331d118a56c65e2460759aeb7cbe3a92a6080b79: Status 404 returned error can't find the container with id ee041b160d660ef171b3245c331d118a56c65e2460759aeb7cbe3a92a6080b79
Oct 30 02:40:15.332740 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.332716    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:15.343967 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:15.343947    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960da804_13fa_4a2b_8837_1dd6f8ca83c3.slice/crio-d7f726f259de8926e44f701d594b910c49baac0548c357964ad12baeb43074bc WatchSource:0}: Error finding container d7f726f259de8926e44f701d594b910c49baac0548c357964ad12baeb43074bc: Status 404 returned error can't find the container with id d7f726f259de8926e44f701d594b910c49baac0548c357964ad12baeb43074bc
Oct 30 02:40:15.345269 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.345109    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:15.355675 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:15.355650    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4e2245_df10_43f4_a52a_433997bc0ca4.slice/crio-c588f69323516c04244fb14ee0a1da3b77a4ed46dba03f0f9a234ffef51e0a70 WatchSource:0}: Error finding container c588f69323516c04244fb14ee0a1da3b77a4ed46dba03f0f9a234ffef51e0a70: Status 404 returned error can't find the container with id c588f69323516c04244fb14ee0a1da3b77a4ed46dba03f0f9a234ffef51e0a70
Oct 30 02:40:15.363722 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.363697    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:15.368095 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.367910    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:15.373913 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:15.373889    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec92445_aa69_4865_9d4e_8a1423bf7073.slice/crio-c6cafa3eebe8def2617ace8a02a87b96390eabea12e5eb033fd7cbaabad16a35 WatchSource:0}: Error finding container c6cafa3eebe8def2617ace8a02a87b96390eabea12e5eb033fd7cbaabad16a35: Status 404 returned error can't find the container with id c6cafa3eebe8def2617ace8a02a87b96390eabea12e5eb033fd7cbaabad16a35
Oct 30 02:40:15.379202 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:15.379180    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816aaac5_8df6_402e_9367_a2ea2f8ca686.slice/crio-49c1fe3992694917dd43544066e8dcc194582356b399624e072c76ef961c2ae8 WatchSource:0}: Error finding container 49c1fe3992694917dd43544066e8dcc194582356b399624e072c76ef961c2ae8: Status 404 returned error can't find the container with id 49c1fe3992694917dd43544066e8dcc194582356b399624e072c76ef961c2ae8
Oct 30 02:40:15.685566 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.685450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:15.685731 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.685645    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:15.685731 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.685730    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs podName:2c04bf2f-3e60-442a-8639-10feae0a3f96 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:16.685707019 +0000 UTC m=+4.086278587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs") pod "network-metrics-daemon-pt72w" (UID: "2c04bf2f-3e60-442a-8639-10feae0a3f96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:15.786352 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.786276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:15.786352 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:15.786327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrsj\" (UniqueName: \"kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj\") pod \"network-check-target-tb8xt\" (UID: \"00bc8689-1f9b-40a7-9f19-972395d1bc69\") " pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:15.786566 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.786468    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:15.786566 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.786486    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:15.786566 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.786498    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wzrsj for pod openshift-network-diagnostics/network-check-target-tb8xt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:15.786713 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.786554    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj podName:00bc8689-1f9b-40a7-9f19-972395d1bc69 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:16.786537417 +0000 UTC m=+4.187108983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wzrsj" (UniqueName: "kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj") pod "network-check-target-tb8xt" (UID: "00bc8689-1f9b-40a7-9f19-972395d1bc69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:15.787153 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.787137    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:15.787247 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:15.787178    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret podName:a2bf413f-45b3-450c-b797-4e674e8da8ec nodeName:}" failed. No retries permitted until 2026-10-30 02:40:16.787167748 +0000 UTC m=+4.187739300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret") pod "global-pull-secret-syncer-hzqcm" (UID: "a2bf413f-45b3-450c-b797-4e674e8da8ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:16.015040 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.014927    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:15 +0000 UTC" deadline="2028-04-02 20:55:07.487240844 +0000 UTC"
Oct 30 02:40:16.015040 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.014981    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12498h14m51.472263545s"
Oct 30 02:40:16.129462 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.129345    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbzl" event={"ID":"fa5e5905-3e48-434f-8153-fd40123a614a","Type":"ContainerStarted","Data":"03cfc82835843a9b804eab876a0e38b76a839fa5e125ba261219a303d233214f"}
Oct 30 02:40:16.138337 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.138189    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs" event={"ID":"816aaac5-8df6-402e-9367-a2ea2f8ca686","Type":"ContainerStarted","Data":"49c1fe3992694917dd43544066e8dcc194582356b399624e072c76ef961c2ae8"}
Oct 30 02:40:16.143843 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.143778    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-flkp8" event={"ID":"1ec92445-aa69-4865-9d4e-8a1423bf7073","Type":"ContainerStarted","Data":"c6cafa3eebe8def2617ace8a02a87b96390eabea12e5eb033fd7cbaabad16a35"}
Oct 30 02:40:16.146968 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.146887    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5" event={"ID":"cb4e2245-df10-43f4-a52a-433997bc0ca4","Type":"ContainerStarted","Data":"c588f69323516c04244fb14ee0a1da3b77a4ed46dba03f0f9a234ffef51e0a70"}
Oct 30 02:40:16.153312 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.153279    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4lscq" event={"ID":"960da804-13fa-4a2b-8837-1dd6f8ca83c3","Type":"ContainerStarted","Data":"d7f726f259de8926e44f701d594b910c49baac0548c357964ad12baeb43074bc"}
Oct 30 02:40:16.155954 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.155929    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qvbmn" event={"ID":"5d7330ef-3e60-407f-89ca-0c0c9e38c87d","Type":"ContainerStarted","Data":"e28b70462fc7a8a20c4d9cd368e1b33c4f9182fe166974975b81eb50d2201bde"}
Oct 30 02:40:16.159976 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.159949    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal" event={"ID":"4f18f0dafccedcaa5cd8761c40142410","Type":"ContainerStarted","Data":"52168c727bd24b83922621ea41531fa6c50fce26a97ea64b594dbb2e3b217b3b"}
Oct 30 02:40:16.167944 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.167918    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal" event={"ID":"6521a54728e46ede7b6de87a0d41821f","Type":"ContainerStarted","Data":"370411047ee34b250d3138bf063e0fa9b97bd5798ad5ae523689dd7f38bdf09d"}
Oct 30 02:40:16.172222 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.172200    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dmfvd" event={"ID":"92515440-cc1f-4e4d-b45e-6afaff621108","Type":"ContainerStarted","Data":"ee041b160d660ef171b3245c331d118a56c65e2460759aeb7cbe3a92a6080b79"}
Oct 30 02:40:16.177602 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.177579    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5l2pw" event={"ID":"caeb5235-93eb-4b90-8372-1c59c807485f","Type":"ContainerStarted","Data":"b9c9384fa29d31d5a3ff68f85599ee9ed4e2155eaefaf775e6a6ee97377bc2a7"}
Oct 30 02:40:16.180738 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.180681    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hqwcj" event={"ID":"af778722-0e75-4ddd-9819-d783cfa858e1","Type":"ContainerStarted","Data":"72213ea7d0fd40202430cd7d816217cc4b53a7a5d9dd3ddffc64d809d2d94e35"}
Oct 30 02:40:16.247669 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.247051    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal" podStartSLOduration=2.247033208 podStartE2EDuration="2.247033208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:16.199044912 +0000 UTC m=+3.599616488" watchObservedRunningTime="2026-10-30 02:40:16.247033208 +0000 UTC m=+3.647604785"
Oct 30 02:40:16.692856 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.692751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:16.693039 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:16.692984    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:16.693098 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:16.693064    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs podName:2c04bf2f-3e60-442a-8639-10feae0a3f96 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:18.69304576 +0000 UTC m=+6.093617317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs") pod "network-metrics-daemon-pt72w" (UID: "2c04bf2f-3e60-442a-8639-10feae0a3f96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:16.793336 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.793303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:16.793503 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:16.793348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrsj\" (UniqueName: \"kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj\") pod \"network-check-target-tb8xt\" (UID: \"00bc8689-1f9b-40a7-9f19-972395d1bc69\") " pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:16.793503 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:16.793484    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:16.793503 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:16.793499    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:16.793651 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:16.793512    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wzrsj for pod openshift-network-diagnostics/network-check-target-tb8xt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:16.793651 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:16.793562    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj podName:00bc8689-1f9b-40a7-9f19-972395d1bc69 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:18.793544707 +0000 UTC m=+6.194116283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wzrsj" (UniqueName: "kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj") pod "network-check-target-tb8xt" (UID: "00bc8689-1f9b-40a7-9f19-972395d1bc69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:16.793951 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:16.793934    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:16.794018 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:16.793987    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret podName:a2bf413f-45b3-450c-b797-4e674e8da8ec nodeName:}" failed. No retries permitted until 2026-10-30 02:40:18.793972797 +0000 UTC m=+6.194544363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret") pod "global-pull-secret-syncer-hzqcm" (UID: "a2bf413f-45b3-450c-b797-4e674e8da8ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:17.016170 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:17.016068    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:15 +0000 UTC" deadline="2028-04-17 11:10:28.12485048 +0000 UTC"
Oct 30 02:40:17.016170 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:17.016101    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12848h30m11.108753641s"
Oct 30 02:40:17.110656 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:17.110624    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:17.110801 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:17.110746    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:17.111170 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:17.111151    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:17.111265 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:17.111246    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:17.111404 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:17.111389    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:17.111523 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:17.111482    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:17.195309 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:17.195273    2576 generic.go:356] "Generic (PLEG): container finished" podID="6521a54728e46ede7b6de87a0d41821f" containerID="370411047ee34b250d3138bf063e0fa9b97bd5798ad5ae523689dd7f38bdf09d" exitCode=0
Oct 30 02:40:17.196320 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:17.196291    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal" event={"ID":"6521a54728e46ede7b6de87a0d41821f","Type":"ContainerDied","Data":"370411047ee34b250d3138bf063e0fa9b97bd5798ad5ae523689dd7f38bdf09d"}
Oct 30 02:40:17.196443 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:17.196345    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal" event={"ID":"6521a54728e46ede7b6de87a0d41821f","Type":"ContainerStarted","Data":"e4fa9b4a42b12ff66b73ed142e26c793aeea6937afad8fae41a2f68f83b12043"}
Oct 30 02:40:18.215050 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:18.213678    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="60aae2bd1ce494349d5ca1f43ada934fb6927531777bdd062b77196a7686481d" exitCode=0
Oct 30 02:40:18.215050 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:18.214738    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbzl" event={"ID":"fa5e5905-3e48-434f-8153-fd40123a614a","Type":"ContainerDied","Data":"60aae2bd1ce494349d5ca1f43ada934fb6927531777bdd062b77196a7686481d"}
Oct 30 02:40:18.282111 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:18.282058    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal" podStartSLOduration=4.28204102 podStartE2EDuration="4.28204102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:17.236826017 +0000 UTC m=+4.637397592" watchObservedRunningTime="2026-10-30 02:40:18.28204102 +0000 UTC m=+5.682612595"
Oct 30 02:40:18.711169 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:18.711093    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:18.711331 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:18.711277    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:18.711387 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:18.711334    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs podName:2c04bf2f-3e60-442a-8639-10feae0a3f96 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:22.711318852 +0000 UTC m=+10.111890408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs") pod "network-metrics-daemon-pt72w" (UID: "2c04bf2f-3e60-442a-8639-10feae0a3f96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:18.812375 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:18.812338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrsj\" (UniqueName: \"kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj\") pod \"network-check-target-tb8xt\" (UID: \"00bc8689-1f9b-40a7-9f19-972395d1bc69\") " pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:18.812539 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:18.812457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:18.812539 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:18.812524    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:18.812662 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:18.812540    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:18.812662 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:18.812595    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret podName:a2bf413f-45b3-450c-b797-4e674e8da8ec nodeName:}" failed. No retries permitted until 2026-10-30 02:40:22.812578202 +0000 UTC m=+10.213149753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret") pod "global-pull-secret-syncer-hzqcm" (UID: "a2bf413f-45b3-450c-b797-4e674e8da8ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:18.812662 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:18.812542    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:18.812662 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:18.812630    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wzrsj for pod openshift-network-diagnostics/network-check-target-tb8xt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:18.812884 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:18.812669    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj podName:00bc8689-1f9b-40a7-9f19-972395d1bc69 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:22.812659073 +0000 UTC m=+10.213230642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wzrsj" (UniqueName: "kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj") pod "network-check-target-tb8xt" (UID: "00bc8689-1f9b-40a7-9f19-972395d1bc69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:19.112069 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:19.112001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:19.112197 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:19.112114    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:19.112441 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:19.112425    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:19.112514 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:19.112498    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:19.112671 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:19.112651    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:19.112786 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:19.112758    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:20.230224 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:20.230145    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hqwcj" event={"ID":"af778722-0e75-4ddd-9819-d783cfa858e1","Type":"ContainerStarted","Data":"e5d4fa458c75aebc30da8032efbd94933d51b2be789fbce2c8df5904ddbec1b9"}
Oct 30 02:40:20.238056 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:20.238026    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5l2pw" event={"ID":"caeb5235-93eb-4b90-8372-1c59c807485f","Type":"ContainerStarted","Data":"95b295cd89b51098c02914e2264409a51a4061aee407f427d2a081739659b01b"}
Oct 30 02:40:20.309509 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:20.309453    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hqwcj" podStartSLOduration=2.665616955 podStartE2EDuration="7.309433542s" totalImagesPullingTime="4.643816587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:20.270549439 +0000 UTC m=+7.671121014" watchObservedRunningTime="2026-10-30 02:40:20.309433542 +0000 UTC m=+7.710005118"
Oct 30 02:40:20.309687 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:20.309560    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5l2pw" podStartSLOduration=3.41607139 podStartE2EDuration="7.309553347s" totalImagesPullingTime="3.893481957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:20.309041241 +0000 UTC m=+7.709612816" watchObservedRunningTime="2026-10-30 02:40:20.309553347 +0000 UTC m=+7.710124921"
Oct 30 02:40:20.915880 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:20.915834    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:20.916433 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:20.916412    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:21.112990 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:21.112474    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:21.112990 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:21.112564    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:21.112990 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:21.112837    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:21.112990 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:21.112936    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:21.113298 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:21.112996    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:21.113298 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:21.113113    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:21.245099 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:21.244626    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5" event={"ID":"cb4e2245-df10-43f4-a52a-433997bc0ca4","Type":"ContainerStarted","Data":"1eb4a271cc0f8815b893a9a50e6d8e3e414391156a3c7b0462cbe4a26c4c3e5f"}
Oct 30 02:40:22.247045 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:22.247010    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:40:22.457932 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:22.457911    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:22.458471 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:22.458453    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:22.464220 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:22.464187    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:40:22.748934 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:22.748904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:22.749097 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:22.749077    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:22.749162 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:22.749134    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs podName:2c04bf2f-3e60-442a-8639-10feae0a3f96 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.749116974 +0000 UTC m=+18.149688541 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs") pod "network-metrics-daemon-pt72w" (UID: "2c04bf2f-3e60-442a-8639-10feae0a3f96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:22.849471 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:22.849286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:22.849471 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:22.849317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrsj\" (UniqueName: \"kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj\") pod \"network-check-target-tb8xt\" (UID: \"00bc8689-1f9b-40a7-9f19-972395d1bc69\") " pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:22.849471 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:22.849447    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:22.849471 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:22.849464    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:22.849612 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:22.849477    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wzrsj for pod openshift-network-diagnostics/network-check-target-tb8xt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:22.849612 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:22.849514    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:22.849612 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:22.849533    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj podName:00bc8689-1f9b-40a7-9f19-972395d1bc69 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.849515571 +0000 UTC m=+18.250087127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wzrsj" (UniqueName: "kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj") pod "network-check-target-tb8xt" (UID: "00bc8689-1f9b-40a7-9f19-972395d1bc69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:22.849612 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:22.849557    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret podName:a2bf413f-45b3-450c-b797-4e674e8da8ec nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.849546139 +0000 UTC m=+18.250117707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret") pod "global-pull-secret-syncer-hzqcm" (UID: "a2bf413f-45b3-450c-b797-4e674e8da8ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:23.042862 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.038621    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:40:22.46419879Z","UUID":"b691c1be-c1de-4268-b894-5ea991769005","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:40:23.042862 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.040762    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:40:23.042862 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.040811    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:40:23.112904 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.110371    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:23.112904 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:23.110475    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:23.112904 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.110526    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:23.112904 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:23.110578    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:23.112904 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.110632    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:23.112904 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:23.110701    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:23.256131 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.256088    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5" event={"ID":"cb4e2245-df10-43f4-a52a-433997bc0ca4","Type":"ContainerStarted","Data":"f5db1d7ca207b52dcaebb1445637197e8731d26c10a5fe0de1f588ec05370e8e"}
Oct 30 02:40:23.258763 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.258731    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qvbmn" event={"ID":"5d7330ef-3e60-407f-89ca-0c0c9e38c87d","Type":"ContainerStarted","Data":"6ef383fb18914115a166eac2a4afdeb1bfac955048a1e7dd599ba21e2b4ca541"}
Oct 30 02:40:23.260535 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.260507    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dmfvd" event={"ID":"92515440-cc1f-4e4d-b45e-6afaff621108","Type":"ContainerStarted","Data":"d34ab2b8746afd8dd2a6cb9c766420ea0a4dcaba60d12d555ff37111a5411be0"}
Oct 30 02:40:23.262788 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.262759    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="d484f28209b3bf91016b3e211aceeeb6d6bda9bde6dbcef8d51d1314395f7a50" exitCode=0
Oct 30 02:40:23.263620 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.263597    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbzl" event={"ID":"fa5e5905-3e48-434f-8153-fd40123a614a","Type":"ContainerDied","Data":"d484f28209b3bf91016b3e211aceeeb6d6bda9bde6dbcef8d51d1314395f7a50"}
Oct 30 02:40:23.339658 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.339605    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qvbmn" podStartSLOduration=2.676369906 podStartE2EDuration="10.339585673s" totalImagesPullingTime="7.663215767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:23.298341105 +0000 UTC m=+10.698912679" watchObservedRunningTime="2026-10-30 02:40:23.339585673 +0000 UTC m=+10.740157247"
Oct 30 02:40:23.340433 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:23.340391    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dmfvd" podStartSLOduration=2.564716198 podStartE2EDuration="10.340378199s" totalImagesPullingTime="7.775662001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:23.33907616 +0000 UTC m=+10.739647734" watchObservedRunningTime="2026-10-30 02:40:23.340378199 +0000 UTC m=+10.740949774"
Oct 30 02:40:24.267445 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:24.267164    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="66617e43942161109e8464ff21a4b3cecfa6e2de6e9ebbd342bd19d2f6fa2a3d" exitCode=0
Oct 30 02:40:24.267818 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:24.267227    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbzl" event={"ID":"fa5e5905-3e48-434f-8153-fd40123a614a","Type":"ContainerDied","Data":"66617e43942161109e8464ff21a4b3cecfa6e2de6e9ebbd342bd19d2f6fa2a3d"}
Oct 30 02:40:24.270801 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:24.270661    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5" event={"ID":"cb4e2245-df10-43f4-a52a-433997bc0ca4","Type":"ContainerStarted","Data":"ba575bda7324af485865181014d480e67b2227c413a4fc3c2ec3fbc2f343a593"}
Oct 30 02:40:24.366126 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:24.366080    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5" podStartSLOduration=3.574746835 podStartE2EDuration="11.366067549s" totalImagesPullingTime="7.791320714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:24.365760689 +0000 UTC m=+11.766332264" watchObservedRunningTime="2026-10-30 02:40:24.366067549 +0000 UTC m=+11.766639122"
Oct 30 02:40:25.110096 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:25.110073    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:25.110210 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:25.110185    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:25.110281 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:25.110256    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:25.110350 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:25.110329    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:25.110437 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:25.110387    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:25.110487 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:25.110464    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:25.273976 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:25.273943    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-flkp8" event={"ID":"1ec92445-aa69-4865-9d4e-8a1423bf7073","Type":"ContainerStarted","Data":"38be4b530e1f525e3360f899bf760f332224d3c377babe14eaa4e2752ba9748b"}
Oct 30 02:40:25.275598 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:25.275572    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4lscq" event={"ID":"960da804-13fa-4a2b-8837-1dd6f8ca83c3","Type":"ContainerStarted","Data":"d14d4d1a2558eef3881eb874271847e8f1268279e147d8acb25107323e09b0a8"}
Oct 30 02:40:25.278243 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:25.278220    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="b02e3b09210c9252795d52a7669f09bf2c273d7bd18ac1ece2aabb20826f64e8" exitCode=0
Oct 30 02:40:25.278354 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:25.278311    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbzl" event={"ID":"fa5e5905-3e48-434f-8153-fd40123a614a","Type":"ContainerDied","Data":"b02e3b09210c9252795d52a7669f09bf2c273d7bd18ac1ece2aabb20826f64e8"}
Oct 30 02:40:25.315264 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:25.315221    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-flkp8" podStartSLOduration=2.649779176 podStartE2EDuration="12.315206442s" totalImagesPullingTime="9.665427266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:25.315190778 +0000 UTC m=+12.715762347" watchObservedRunningTime="2026-10-30 02:40:25.315206442 +0000 UTC m=+12.715778015"
Oct 30 02:40:25.398352 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:25.398299    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4lscq" podStartSLOduration=4.477940395 podStartE2EDuration="12.398283327s" totalImagesPullingTime="7.920342932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:25.397890403 +0000 UTC m=+12.798461975" watchObservedRunningTime="2026-10-30 02:40:25.398283327 +0000 UTC m=+12.798854900"
Oct 30 02:40:27.109747 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:27.109724    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:27.110228 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:27.109724    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:27.110228 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:27.109866    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:27.110228 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:27.109725    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:27.110228 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:27.109943    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:27.110228 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:27.110043    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:27.284891 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:27.284799    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="f4f4d19614e511fe50e77cbe55ea371649eba756aef1307273b08ea9b5964a8e" exitCode=0
Oct 30 02:40:27.284891 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:27.284881    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbzl" event={"ID":"fa5e5905-3e48-434f-8153-fd40123a614a","Type":"ContainerDied","Data":"f4f4d19614e511fe50e77cbe55ea371649eba756aef1307273b08ea9b5964a8e"}
Oct 30 02:40:28.290811 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:28.290586    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="a4b2d7f9c00b257b2eb6daf2576131d1b06096d8851081b339aeee6d1a048f97" exitCode=0
Oct 30 02:40:28.291348 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:28.290672    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbzl" event={"ID":"fa5e5905-3e48-434f-8153-fd40123a614a","Type":"ContainerDied","Data":"a4b2d7f9c00b257b2eb6daf2576131d1b06096d8851081b339aeee6d1a048f97"}
Oct 30 02:40:29.110062 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:29.110027    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:29.110062 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:29.110052    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:29.110245 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:29.110204    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:29.110245 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:29.110235    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:29.110327 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:29.110310    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:29.110423 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:29.110402    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:29.295961 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:29.295927    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbzl" event={"ID":"fa5e5905-3e48-434f-8153-fd40123a614a","Type":"ContainerStarted","Data":"1b5d491658b5dc35c720eef36b4194a747af3b9ba11a173d4d28ec1e88b72dd4"}
Oct 30 02:40:29.364309 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:29.364230    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8pbzl" podStartSLOduration=6.71918968 podStartE2EDuration="16.364218063s" totalImagesPullingTime="9.645028383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:29.364055367 +0000 UTC m=+16.764626939" watchObservedRunningTime="2026-10-30 02:40:29.364218063 +0000 UTC m=+16.764789636"
Oct 30 02:40:30.300534 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:30.300366    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:40:30.300946 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:30.300815    2576 generic.go:356] "Generic (PLEG): container finished" podID="816aaac5-8df6-402e-9367-a2ea2f8ca686" containerID="5651f2e0747b4afda8b7cafc7fa07f94b759a8fae4574fc59b5ae22bf7339f80" exitCode=1
Oct 30 02:40:30.300946 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:30.300874    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs" event={"ID":"816aaac5-8df6-402e-9367-a2ea2f8ca686","Type":"ContainerStarted","Data":"5815b99b92be62a299f4be994b3f8d1b135ca4b277b6ed183a7140055b258989"}
Oct 30 02:40:30.300946 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:30.300931    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs" event={"ID":"816aaac5-8df6-402e-9367-a2ea2f8ca686","Type":"ContainerStarted","Data":"5d5d4aec194f64ab5cf3ac2edbc522b7ba8ec2edc9a2d329e067656844be670d"}
Oct 30 02:40:30.301051 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:30.300960    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs" event={"ID":"816aaac5-8df6-402e-9367-a2ea2f8ca686","Type":"ContainerStarted","Data":"26fb94f480e0cfa060b1defed70cba6fcc6c94123c9461644614fd841e319781"}
Oct 30 02:40:30.301051 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:30.300987    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs" event={"ID":"816aaac5-8df6-402e-9367-a2ea2f8ca686","Type":"ContainerStarted","Data":"2d3f1aaa176a2447aa3115f6c6079e7a02ea34aa14322672457e1800f5410a14"}
Oct 30 02:40:30.301051 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:30.301014    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs" event={"ID":"816aaac5-8df6-402e-9367-a2ea2f8ca686","Type":"ContainerDied","Data":"5651f2e0747b4afda8b7cafc7fa07f94b759a8fae4574fc59b5ae22bf7339f80"}
Oct 30 02:40:30.301051 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:30.301044    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs" event={"ID":"816aaac5-8df6-402e-9367-a2ea2f8ca686","Type":"ContainerStarted","Data":"693543519bccf0207c102d3610c82ca1d1030eee07bae1b12bf3cb75d9644637"}
Oct 30 02:40:30.811703 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:30.811672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:30.811900 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:30.811818    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:30.811900 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:30.811892    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs podName:2c04bf2f-3e60-442a-8639-10feae0a3f96 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.811876253 +0000 UTC m=+34.212447815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs") pod "network-metrics-daemon-pt72w" (UID: "2c04bf2f-3e60-442a-8639-10feae0a3f96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:30.912506 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:30.912478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:30.912506 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:30.912507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrsj\" (UniqueName: \"kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj\") pod \"network-check-target-tb8xt\" (UID: \"00bc8689-1f9b-40a7-9f19-972395d1bc69\") " pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:30.912650 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:30.912600    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:30.912650 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:30.912606    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:30.912703 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:30.912658    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret podName:a2bf413f-45b3-450c-b797-4e674e8da8ec nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.912645609 +0000 UTC m=+34.313217163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret") pod "global-pull-secret-syncer-hzqcm" (UID: "a2bf413f-45b3-450c-b797-4e674e8da8ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:30.912703 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:30.912614    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:30.912783 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:30.912704    2576 projected.go:196] Error preparing data for projected volume kube-api-access-wzrsj for pod openshift-network-diagnostics/network-check-target-tb8xt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:30.912783 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:30.912748    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj podName:00bc8689-1f9b-40a7-9f19-972395d1bc69 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.912732828 +0000 UTC m=+34.313304395 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wzrsj" (UniqueName: "kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj") pod "network-check-target-tb8xt" (UID: "00bc8689-1f9b-40a7-9f19-972395d1bc69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:31.110620 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:31.110552    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:31.110620 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:31.110568    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:31.110771 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:31.110658    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:31.110771 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:31.110715    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:31.110771 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:31.110753    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:31.110896 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:31.110812    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:32.307076 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:32.307054    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:40:32.307514 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:32.307398    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs" event={"ID":"816aaac5-8df6-402e-9367-a2ea2f8ca686","Type":"ContainerStarted","Data":"0bb0b741cb06784f0568b19ec892620489ec57dfd1457098aa63c0cb092ceb8d"}
Oct 30 02:40:33.110611 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:33.110581    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:33.110734 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:33.110670    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:33.110774 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:33.110759    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:33.110887 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:33.110869    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:33.110943 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:33.110922    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:33.111007 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:33.110988    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:34.314550 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:34.314521    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:40:34.315220 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:34.314891    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs" event={"ID":"816aaac5-8df6-402e-9367-a2ea2f8ca686","Type":"ContainerStarted","Data":"0be1c3f0e085c30e05b485b5e09c66155d7c3ecb7aaf1ccf0b7cff80a8b64ca0"}
Oct 30 02:40:34.315297 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:34.315237    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:34.315625 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:34.315320    2576 scope.go:122] "RemoveContainer" containerID="5651f2e0747b4afda8b7cafc7fa07f94b759a8fae4574fc59b5ae22bf7339f80"
Oct 30 02:40:34.315819 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:34.315805    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:34.331055 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:34.331036    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:35.110297 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:35.110273    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:35.110457 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:35.110306    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:35.110457 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:35.110382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:35.110518 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:35.110488    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:35.110615 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:35.110599    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:35.110669 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:35.110655    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:35.322475 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:35.322443    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:40:35.322953 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:35.322894    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs" event={"ID":"816aaac5-8df6-402e-9367-a2ea2f8ca686","Type":"ContainerStarted","Data":"eb7fdcbdb2e2f9b0609fa02fcde9472f4f23dcbbaec70018997996ac1e6fd6ae"}
Oct 30 02:40:35.324072 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:35.324055    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:35.339593 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:35.339575    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:35.414347 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:35.414265    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs" podStartSLOduration=8.407059935 podStartE2EDuration="22.414253337s" totalImagesPullingTime="14.007193402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:35.401324026 +0000 UTC m=+22.801895599" watchObservedRunningTime="2026-10-30 02:40:35.414253337 +0000 UTC m=+22.814824909"
Oct 30 02:40:36.039692 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:36.039662    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hzqcm"]
Oct 30 02:40:36.039833 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:36.039755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:36.039914 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:36.039832    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:36.042745 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:36.042720    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tb8xt"]
Oct 30 02:40:36.042843 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:36.042815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:36.042963 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:36.042939    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:36.046481 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:36.046461    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pt72w"]
Oct 30 02:40:36.046548 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:36.046539    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:36.046641 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:36.046619    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:37.110327 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:37.110296    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:37.110869 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:37.110296    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:37.110869 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:37.110413    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:37.110869 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:37.110307    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:37.110869 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:37.110466    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:37.110869 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:37.110546    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:39.110130 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:39.110098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:39.110130 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:39.110126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:39.110605 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:39.110098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:39.110605 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:39.110196    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:39.110605 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:39.110279    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:39.110605 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:39.110353    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:41.110111 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:41.110085    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:41.110111 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:41.110105    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:41.110558 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:41.110227    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pt72w" podUID="2c04bf2f-3e60-442a-8639-10feae0a3f96"
Oct 30 02:40:41.110558 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:41.110278    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:41.110558 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:41.110340    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hzqcm" podUID="a2bf413f-45b3-450c-b797-4e674e8da8ec"
Oct 30 02:40:41.110558 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:41.110414    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8xt" podUID="00bc8689-1f9b-40a7-9f19-972395d1bc69"
Oct 30 02:40:42.896129 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:42.896102    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeReady"
Oct 30 02:40:42.896488 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:42.896195    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 02:40:42.962241 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:42.962207    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cvtzh"]
Oct 30 02:40:43.008549 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.008522    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"]
Oct 30 02:40:43.009871 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.009804    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cvtzh"
Oct 30 02:40:43.021806 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.021776    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6b959b9795-jw4rz"]
Oct 30 02:40:43.022061 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.021975    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:43.032492 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.030768    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 02:40:43.032492 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.030881    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 02:40:43.039973 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.039951    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"]
Oct 30 02:40:43.040093 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.040048    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.041257 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.041237    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 02:40:43.057678 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.057656    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j86rx"]
Oct 30 02:40:43.057811 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.057794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:43.066877 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.066843    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"]
Oct 30 02:40:43.066977 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.066937    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j86rx"
Oct 30 02:40:43.078678 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.078658    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-888jp"]
Oct 30 02:40:43.078765 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.078744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:43.101153 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.101134    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw"]
Oct 30 02:40:43.101230 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.101202    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.101303 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.101286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.101351 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.101320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghvzs\" (UniqueName: \"kubernetes.io/projected/bf8a2708-a7fe-41b0-8f63-7850df383c8f-kube-api-access-ghvzs\") pod \"kas-connection-checker-685895c7c9-cvtzh\" (UID: \"bf8a2708-a7fe-41b0-8f63-7850df383c8f\") " pod="kube-system/kas-connection-checker-685895c7c9-cvtzh"
Oct 30 02:40:43.101401 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.101349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.101449 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.101415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:43.101571 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.101550    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-default-certificate\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.101655 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.101603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-stats-auth\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.101703 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.101656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzh75\" (UniqueName: \"kubernetes.io/projected/6c28bc38-2c97-412f-89b0-f7a552b78373-kube-api-access-lzh75\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.101745 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.101681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:43.101745 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.101718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p5wx\" (UniqueName: \"kubernetes.io/projected/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-kube-api-access-6p5wx\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:43.109725 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.109708    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw"
Oct 30 02:40:43.112526 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.112508    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"]
Oct 30 02:40:43.112607 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.112588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:43.112607 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.112595    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:43.112687 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.112586    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:43.126767 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.126742    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 02:40:43.126882 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.126872    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 02:40:43.126957 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.126899    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:40:43.127014 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.126961    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:40:43.127014 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.126965    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 02:40:43.127014 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.126991    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:43.127127 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.127011    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:43.127207 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.127194    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:43.127378 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.127362    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 02:40:43.127435 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.127375    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 02:40:43.127483 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.127470    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:43.127524 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.127487    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 02:40:43.127557 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.127527    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:43.127557 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.127529    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:43.127557 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.127532    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:43.127649 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.127536    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:43.131475 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.131449    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zh9rv"]
Oct 30 02:40:43.131563 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.131550    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:43.139700 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.139683    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 02:40:43.140769 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.140739    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:40:43.140874 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.140777    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:40:43.144577 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.142508    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:40:43.144577 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.142780    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 02:40:43.144577 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.143016    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 02:40:43.144577 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.143213    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 02:40:43.144577 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.143309    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 02:40:43.144577 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.143505    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 02:40:43.144577 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.143307    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kk2sv\""
Oct 30 02:40:43.144577 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.143686    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qc2sc\""
Oct 30 02:40:43.144577 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.143789    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:40:43.146011 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.145993    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 02:40:43.148708 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.148656    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67544b9897-hb2j2"]
Oct 30 02:40:43.148797 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.148773    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.151763 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.151745    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 02:40:43.151833 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.151779    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xsmz\""
Oct 30 02:40:43.169275 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.169252    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sj78j"]
Oct 30 02:40:43.169363 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.169354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.185643 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.185627    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9"]
Oct 30 02:40:43.185736 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.185723    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:43.193069 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.193044    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56b7889b4-hzhbm"]
Oct 30 02:40:43.193161 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.193137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9"
Oct 30 02:40:43.202172 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202153    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04ea065c-0b57-43ec-99be-989bb703b2c5-serving-cert\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:43.202263 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:43.202263 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgnv\" (UniqueName: \"kubernetes.io/projected/1ea0be63-7705-40e8-b2a1-e48c39259c8b-kube-api-access-pvgnv\") pod \"kas-connection-checker-685895c7c9-j86rx\" (UID: \"1ea0be63-7705-40e8-b2a1-e48c39259c8b\") " pod="kube-system/kas-connection-checker-685895c7c9-j86rx"
Oct 30 02:40:43.202263 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-trusted-ca\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.202385 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghvzs\" (UniqueName: \"kubernetes.io/projected/bf8a2708-a7fe-41b0-8f63-7850df383c8f-kube-api-access-ghvzs\") pod \"kas-connection-checker-685895c7c9-cvtzh\" (UID: \"bf8a2708-a7fe-41b0-8f63-7850df383c8f\") " pod="kube-system/kas-connection-checker-685895c7c9-cvtzh"
Oct 30 02:40:43.202385 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjbq\" (UniqueName: \"kubernetes.io/projected/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-kube-api-access-ptjbq\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:43.202484 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbq6\" (UniqueName: \"kubernetes.io/projected/2967919e-d201-48ae-b388-85fe023e1d21-kube-api-access-rmbq6\") pod \"volume-data-source-validator-58c9f5f46d-cd7dw\" (UID: \"2967919e-d201-48ae-b388-85fe023e1d21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw"
Oct 30 02:40:43.202484 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b6386c9-d170-4b89-bb41-9a0641207188-serving-cert\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.202583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.202583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:43.202583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202554    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:43.202724 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202581    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:43.202724 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.202597    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.702584281 +0000 UTC m=+31.103155832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : configmap references non-existent config key: service-ca.crt
Oct 30 02:40:43.202724 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:43.202724 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04ea065c-0b57-43ec-99be-989bb703b2c5-config\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:43.202724 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6qd\" (UniqueName: \"kubernetes.io/projected/04ea065c-0b57-43ec-99be-989bb703b2c5-kube-api-access-dw6qd\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:43.203013 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-config\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.203013 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7cpv\" (UniqueName: \"kubernetes.io/projected/7b6386c9-d170-4b89-bb41-9a0641207188-kube-api-access-d7cpv\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.203013 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-default-certificate\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.203013 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-stats-auth\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.203013 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzh75\" (UniqueName: \"kubernetes.io/projected/6c28bc38-2c97-412f-89b0-f7a552b78373-kube-api-access-lzh75\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.203013 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:43.203013 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5wx\" (UniqueName: \"kubernetes.io/projected/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-kube-api-access-6p5wx\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:43.203013 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202917    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x9rq\" (UniqueName: \"kubernetes.io/projected/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-kube-api-access-9x9rq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:43.203013 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.202935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.203013 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.203004    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:40:43.203430 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.203036    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.703025748 +0000 UTC m=+31.103597298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : secret "router-metrics-certs-default" not found
Oct 30 02:40:43.203430 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.203074    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:43.203430 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.203093    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls podName:c9aa32ac-0748-4d2f-820d-8a15ec78cdb1 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.703087276 +0000 UTC m=+31.103658826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gl8sd" (UID: "c9aa32ac-0748-4d2f-820d-8a15ec78cdb1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:43.203430 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.203341    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:43.206231 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.206213    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-default-certificate\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.206231 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.206222    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-stats-auth\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.217068 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.217051    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7btgh"]
Oct 30 02:40:43.217143 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.217057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.225563 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.225540    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghvzs\" (UniqueName: \"kubernetes.io/projected/bf8a2708-a7fe-41b0-8f63-7850df383c8f-kube-api-access-ghvzs\") pod \"kas-connection-checker-685895c7c9-cvtzh\" (UID: \"bf8a2708-a7fe-41b0-8f63-7850df383c8f\") " pod="kube-system/kas-connection-checker-685895c7c9-cvtzh"
Oct 30 02:40:43.226188 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.226160    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:40:43.226188 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.226169    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:40:43.226188 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.226188    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 02:40:43.226381 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.226245    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:43.226381 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.226278    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 02:40:43.226497 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.226445    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5wx\" (UniqueName: \"kubernetes.io/projected/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-kube-api-access-6p5wx\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:43.226553 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.226531    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:43.226604 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.226575    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:40:43.227256 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.227237    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzh75\" (UniqueName: \"kubernetes.io/projected/6c28bc38-2c97-412f-89b0-f7a552b78373-kube-api-access-lzh75\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.228081 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.228065    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 02:40:43.228129 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.228065    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:40:43.231081 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.231066    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m9rxl"]
Oct 30 02:40:43.231191 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.231177    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7btgh"
Oct 30 02:40:43.241140 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.241122    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 02:40:43.241603 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.241585    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 02:40:43.241603 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.241588    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 02:40:43.241701 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.241608    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 02:40:43.241701 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.241592    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 02:40:43.241701 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.241591    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 02:40:43.241701 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.241643    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fd755\""
Oct 30 02:40:43.241701 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.241658    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-zzvjn\""
Oct 30 02:40:43.243470 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.243457    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9r4jn"]
Oct 30 02:40:43.243561 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.243549    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.253610 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253592    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cvtzh"]
Oct 30 02:40:43.253700 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253620    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"]
Oct 30 02:40:43.253700 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253641    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"]
Oct 30 02:40:43.253700 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253668    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b959b9795-jw4rz"]
Oct 30 02:40:43.253700 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253689    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"]
Oct 30 02:40:43.253882 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253707    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"]
Oct 30 02:40:43.253882 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:43.253882 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253727    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-888jp"]
Oct 30 02:40:43.253882 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253747    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7btgh"]
Oct 30 02:40:43.253882 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253768    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9"]
Oct 30 02:40:43.253882 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253789    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sj78j"]
Oct 30 02:40:43.253882 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253807    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m9rxl"]
Oct 30 02:40:43.253882 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253833    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j86rx"]
Oct 30 02:40:43.253882 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253872    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9r4jn"]
Oct 30 02:40:43.254244 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253891    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw"]
Oct 30 02:40:43.254244 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253911    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zh9rv"]
Oct 30 02:40:43.254244 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253930    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67544b9897-hb2j2"]
Oct 30 02:40:43.254244 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.253947    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56b7889b4-hzhbm"]
Oct 30 02:40:43.304138 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt4f4\" (UniqueName: \"kubernetes.io/projected/16395f11-a919-4dd3-9d21-3388d472e616-kube-api-access-dt4f4\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.304250 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-image-registry-private-configuration\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.304250 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbq6\" (UniqueName: \"kubernetes.io/projected/2967919e-d201-48ae-b388-85fe023e1d21-kube-api-access-rmbq6\") pod \"volume-data-source-validator-58c9f5f46d-cd7dw\" (UID: \"2967919e-d201-48ae-b388-85fe023e1d21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw"
Oct 30 02:40:43.304250 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304226    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.304376 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-trusted-ca\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.304443 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kk4c\" (UniqueName: \"kubernetes.io/projected/76243024-183c-42e6-89fd-ccd82d3571c4-kube-api-access-5kk4c\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.304486 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-installation-pull-secrets\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.304558 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304539    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:43.304604 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgnv\" (UniqueName: \"kubernetes.io/projected/1ea0be63-7705-40e8-b2a1-e48c39259c8b-kube-api-access-pvgnv\") pod \"kas-connection-checker-685895c7c9-j86rx\" (UID: \"1ea0be63-7705-40e8-b2a1-e48c39259c8b\") " pod="kube-system/kas-connection-checker-685895c7c9-j86rx"
Oct 30 02:40:43.304656 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs586\" (UniqueName: \"kubernetes.io/projected/51189dec-b0c6-430e-9768-8f9532bdb22a-kube-api-access-cs586\") pod \"network-check-source-6f78d59f99-tngw9\" (UID: \"51189dec-b0c6-430e-9768-8f9532bdb22a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9"
Oct 30 02:40:43.304656 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.304756 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd57c8d3-5a2c-4068-b04a-901e5795b815-ca-trust-extracted\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.304756 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304702    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfg6j\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-kube-api-access-dfg6j\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.304756 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-tmp\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.304756 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526jr\" (UniqueName: \"kubernetes.io/projected/4844aa91-37e4-4cae-a380-af0000c768fc-kube-api-access-526jr\") pod \"kas-connection-checker-685895c7c9-7btgh\" (UID: \"4844aa91-37e4-4cae-a380-af0000c768fc\") " pod="kube-system/kas-connection-checker-685895c7c9-7btgh"
Oct 30 02:40:43.304983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304787    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:43.304983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16395f11-a919-4dd3-9d21-3388d472e616-config-volume\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.304983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304896    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-config\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.304983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-certificates\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.304983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304928    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7cpv\" (UniqueName: \"kubernetes.io/projected/7b6386c9-d170-4b89-bb41-9a0641207188-kube-api-access-d7cpv\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.304983 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.304953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:43.305257 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-trusted-ca\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.305257 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-installation-pull-secrets\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.305257 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b6386c9-d170-4b89-bb41-9a0641207188-serving-cert\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.305257 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-bound-sa-token\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.305257 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305200    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x9rq\" (UniqueName: \"kubernetes.io/projected/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-kube-api-access-9x9rq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:43.305257 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305219    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-trusted-ca\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.305257 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.305257 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305226    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:43.305562 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305285    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.305562 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04ea065c-0b57-43ec-99be-989bb703b2c5-serving-cert\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:43.305562 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305403    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjbq\" (UniqueName: \"kubernetes.io/projected/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-kube-api-access-ptjbq\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:43.305562 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305508    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/69caed51-464d-4dab-b4a4-b4ff77da5e4e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:43.305562 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:43.305769 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305559    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-config\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.305769 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76243024-183c-42e6-89fd-ccd82d3571c4-serving-cert\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.305769 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305644    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-ca-trust-extracted\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.305769 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.305681    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:40:43.305769 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-image-registry-private-configuration\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.305769 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305721    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-certificates\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.305769 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.305772    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls podName:215f867e-a2ba-4b0d-ada3-6182c5e1a8fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.8057549 +0000 UTC m=+31.206326463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pppd8" (UID: "215f867e-a2ba-4b0d-ada3-6182c5e1a8fd") : secret "samples-operator-tls" not found
Oct 30 02:40:43.306120 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:43.306120 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-service-ca-bundle\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.306120 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16395f11-a919-4dd3-9d21-3388d472e616-tmp-dir\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.306120 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04ea065c-0b57-43ec-99be-989bb703b2c5-config\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:43.306120 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.305978    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6qd\" (UniqueName: \"kubernetes.io/projected/04ea065c-0b57-43ec-99be-989bb703b2c5-kube-api-access-dw6qd\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:43.306120 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.306018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-snapshots\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.306120 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.306046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82nmv\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-kube-api-access-82nmv\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.306120 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.306085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-bound-sa-token\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.306500 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.306139    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-trusted-ca\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.306500 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.306381    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:43.306500 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.306451    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04ea065c-0b57-43ec-99be-989bb703b2c5-config\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:43.307361 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.307343    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:43.307499 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.307461    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b6386c9-d170-4b89-bb41-9a0641207188-serving-cert\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.307962 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.307946    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04ea065c-0b57-43ec-99be-989bb703b2c5-serving-cert\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:43.323537 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.323516    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbq6\" (UniqueName: \"kubernetes.io/projected/2967919e-d201-48ae-b388-85fe023e1d21-kube-api-access-rmbq6\") pod \"volume-data-source-validator-58c9f5f46d-cd7dw\" (UID: \"2967919e-d201-48ae-b388-85fe023e1d21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw"
Oct 30 02:40:43.324080 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.324065    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7cpv\" (UniqueName: \"kubernetes.io/projected/7b6386c9-d170-4b89-bb41-9a0641207188-kube-api-access-d7cpv\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.326009 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.325988    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:40:43.326176 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.326159    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:40:43.326266 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.326250    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:40:43.327391 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.327369    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgnv\" (UniqueName: \"kubernetes.io/projected/1ea0be63-7705-40e8-b2a1-e48c39259c8b-kube-api-access-pvgnv\") pod \"kas-connection-checker-685895c7c9-j86rx\" (UID: \"1ea0be63-7705-40e8-b2a1-e48c39259c8b\") " pod="kube-system/kas-connection-checker-685895c7c9-j86rx"
Oct 30 02:40:43.327653 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.327634    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x9rq\" (UniqueName: \"kubernetes.io/projected/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-kube-api-access-9x9rq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:43.327786 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.327768    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjbq\" (UniqueName: \"kubernetes.io/projected/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-kube-api-access-ptjbq\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:43.328346 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.328329    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6qd\" (UniqueName: \"kubernetes.io/projected/04ea065c-0b57-43ec-99be-989bb703b2c5-kube-api-access-dw6qd\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:43.334263 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.334249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cvtzh"
Oct 30 02:40:43.340885 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.340864    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:40:43.340994 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.340980    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:40:43.341054 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.341019    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kdxrj\""
Oct 30 02:40:43.341096 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.341055    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xn2tj\""
Oct 30 02:40:43.406932 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.406872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76243024-183c-42e6-89fd-ccd82d3571c4-serving-cert\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.406932 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.406911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-ca-trust-extracted\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.407099 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.406938    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-image-registry-private-configuration\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.407099 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.406961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-certificates\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.407337 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407319    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-ca-trust-extracted\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.407446 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-service-ca-bundle\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.407499 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16395f11-a919-4dd3-9d21-3388d472e616-tmp-dir\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.407552 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-snapshots\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.407552 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82nmv\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-kube-api-access-82nmv\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.407656 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-bound-sa-token\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.407656 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407581    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-certificates\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.407656 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-trusted-ca\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.407656 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt4f4\" (UniqueName: \"kubernetes.io/projected/16395f11-a919-4dd3-9d21-3388d472e616-kube-api-access-dt4f4\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.407880 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-image-registry-private-configuration\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.407880 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.407880 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407769    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0334aa2-bc83-440a-8781-60817828e1ab-cert\") pod \"ingress-canary-9r4jn\" (UID: \"e0334aa2-bc83-440a-8781-60817828e1ab\") " pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:43.407880 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kk4c\" (UniqueName: \"kubernetes.io/projected/76243024-183c-42e6-89fd-ccd82d3571c4-kube-api-access-5kk4c\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.407880 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407824    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16395f11-a919-4dd3-9d21-3388d472e616-tmp-dir\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.407880 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-installation-pull-secrets\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.408173 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs586\" (UniqueName: \"kubernetes.io/projected/51189dec-b0c6-430e-9768-8f9532bdb22a-kube-api-access-cs586\") pod \"network-check-source-6f78d59f99-tngw9\" (UID: \"51189dec-b0c6-430e-9768-8f9532bdb22a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9"
Oct 30 02:40:43.408173 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.408173 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd57c8d3-5a2c-4068-b04a-901e5795b815-ca-trust-extracted\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.408173 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408018    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-service-ca-bundle\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.408173 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.408115    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:43.408378 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.408181    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls podName:16395f11-a919-4dd3-9d21-3388d472e616 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.908163763 +0000 UTC m=+31.308735330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls") pod "dns-default-m9rxl" (UID: "16395f11-a919-4dd3-9d21-3388d472e616") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:43.408378 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.407994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfg6j\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-kube-api-access-dfg6j\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.408378 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-tmp\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.408378 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526jr\" (UniqueName: \"kubernetes.io/projected/4844aa91-37e4-4cae-a380-af0000c768fc-kube-api-access-526jr\") pod \"kas-connection-checker-685895c7c9-7btgh\" (UID: \"4844aa91-37e4-4cae-a380-af0000c768fc\") " pod="kube-system/kas-connection-checker-685895c7c9-7btgh"
Oct 30 02:40:43.408378 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:43.408378 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16395f11-a919-4dd3-9d21-3388d472e616-config-volume\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.408546 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-certificates\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.408546 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-trusted-ca\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.408546 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-installation-pull-secrets\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.408546 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-bound-sa-token\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.408709 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.408709 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.408709 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408589    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-snapshots\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.408709 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408624    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-trusted-ca\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.408709 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/69caed51-464d-4dab-b4a4-b4ff77da5e4e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:43.408709 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd57c8d3-5a2c-4068-b04a-901e5795b815-ca-trust-extracted\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.408709 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tllq7\" (UniqueName: \"kubernetes.io/projected/e0334aa2-bc83-440a-8781-60817828e1ab-kube-api-access-tllq7\") pod \"ingress-canary-9r4jn\" (UID: \"e0334aa2-bc83-440a-8781-60817828e1ab\") " pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:43.408990 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.408902    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-tmp\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.409359 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.409334    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/69caed51-464d-4dab-b4a4-b4ff77da5e4e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:43.409448 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.409431    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:40:43.409448 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.409448    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56b7889b4-hzhbm: secret "image-registry-tls" not found
Oct 30 02:40:43.409564 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.409483    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-certificates\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.409564 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.409493    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls podName:4f8a405e-f80a-4aa9-ab4c-2217e9250b70 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.909478438 +0000 UTC m=+31.310050006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls") pod "image-registry-56b7889b4-hzhbm" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70") : secret "image-registry-tls" not found
Oct 30 02:40:43.410150 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.409764    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:40:43.410150 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.409807    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16395f11-a919-4dd3-9d21-3388d472e616-config-volume\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.410150 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.409827    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert podName:69caed51-464d-4dab-b4a4-b4ff77da5e4e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.909811483 +0000 UTC m=+31.310383047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sj78j" (UID: "69caed51-464d-4dab-b4a4-b4ff77da5e4e") : secret "networking-console-plugin-cert" not found
Oct 30 02:40:43.410150 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.409910    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:40:43.410150 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.409923    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67544b9897-hb2j2: secret "image-registry-tls" not found
Oct 30 02:40:43.410150 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.409959    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls podName:cd57c8d3-5a2c-4068-b04a-901e5795b815 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.909947607 +0000 UTC m=+31.310519162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls") pod "image-registry-67544b9897-hb2j2" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815") : secret "image-registry-tls" not found
Oct 30 02:40:43.410150 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.409957    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-trusted-ca\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.410578 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.410558    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.411157 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.411133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-image-registry-private-configuration\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.411575 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.411556    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-installation-pull-secrets\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.411967 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.411946    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-image-registry-private-configuration\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.412437 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.412416    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76243024-183c-42e6-89fd-ccd82d3571c4-serving-cert\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.412519 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.412506    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-installation-pull-secrets\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.418628 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.418608    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:43.430550 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.429881    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt4f4\" (UniqueName: \"kubernetes.io/projected/16395f11-a919-4dd3-9d21-3388d472e616-kube-api-access-dt4f4\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.430550 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.430192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-bound-sa-token\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.430550 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.430311    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-526jr\" (UniqueName: \"kubernetes.io/projected/4844aa91-37e4-4cae-a380-af0000c768fc-kube-api-access-526jr\") pod \"kas-connection-checker-685895c7c9-7btgh\" (UID: \"4844aa91-37e4-4cae-a380-af0000c768fc\") " pod="kube-system/kas-connection-checker-685895c7c9-7btgh"
Oct 30 02:40:43.430550 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.430583    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfg6j\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-kube-api-access-dfg6j\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.431111 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.431083    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs586\" (UniqueName: \"kubernetes.io/projected/51189dec-b0c6-430e-9768-8f9532bdb22a-kube-api-access-cs586\") pod \"network-check-source-6f78d59f99-tngw9\" (UID: \"51189dec-b0c6-430e-9768-8f9532bdb22a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9"
Oct 30 02:40:43.431383 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.431365    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-bound-sa-token\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.431383 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.431374    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kk4c\" (UniqueName: \"kubernetes.io/projected/76243024-183c-42e6-89fd-ccd82d3571c4-kube-api-access-5kk4c\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.431523 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.431490    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82nmv\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-kube-api-access-82nmv\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.437413 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.437389    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:43.470378 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.470347    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:43.484167 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.484140    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw"
Oct 30 02:40:43.499939 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.499133    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j86rx"
Oct 30 02:40:43.511033 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.509835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tllq7\" (UniqueName: \"kubernetes.io/projected/e0334aa2-bc83-440a-8781-60817828e1ab-kube-api-access-tllq7\") pod \"ingress-canary-9r4jn\" (UID: \"e0334aa2-bc83-440a-8781-60817828e1ab\") " pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:43.511033 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.509987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0334aa2-bc83-440a-8781-60817828e1ab-cert\") pod \"ingress-canary-9r4jn\" (UID: \"e0334aa2-bc83-440a-8781-60817828e1ab\") " pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:43.515711 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.513028    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0334aa2-bc83-440a-8781-60817828e1ab-cert\") pod \"ingress-canary-9r4jn\" (UID: \"e0334aa2-bc83-440a-8781-60817828e1ab\") " pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:43.534877 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.533524    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tllq7\" (UniqueName: \"kubernetes.io/projected/e0334aa2-bc83-440a-8781-60817828e1ab-kube-api-access-tllq7\") pod \"ingress-canary-9r4jn\" (UID: \"e0334aa2-bc83-440a-8781-60817828e1ab\") " pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:43.538297 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.538271    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:43.578491 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.578277    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9"
Oct 30 02:40:43.583524 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.583501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7btgh"
Oct 30 02:40:43.595693 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.593964    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:43.688279 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.688255    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cvtzh"]
Oct 30 02:40:43.693658 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:43.693625    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8a2708_a7fe_41b0_8f63_7850df383c8f.slice/crio-72dd658f9795830c9a88e19578bdde58deb6ee4df059e5b8b0d8a1b205b11205 WatchSource:0}: Error finding container 72dd658f9795830c9a88e19578bdde58deb6ee4df059e5b8b0d8a1b205b11205: Status 404 returned error can't find the container with id 72dd658f9795830c9a88e19578bdde58deb6ee4df059e5b8b0d8a1b205b11205
Oct 30 02:40:43.726153 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.719984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.726153 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.720118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:43.726153 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.720182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:43.726153 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.720427    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:40:43.726153 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.720499    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.720473316 +0000 UTC m=+32.121044868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : secret "router-metrics-certs-default" not found
Oct 30 02:40:43.726153 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.720600    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.720586597 +0000 UTC m=+32.121158152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : configmap references non-existent config key: service-ca.crt
Oct 30 02:40:43.726153 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.720683    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:43.726153 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.720737    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls podName:c9aa32ac-0748-4d2f-820d-8a15ec78cdb1 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.720715501 +0000 UTC m=+32.121287066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gl8sd" (UID: "c9aa32ac-0748-4d2f-820d-8a15ec78cdb1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:43.737441 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.737413    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"]
Oct 30 02:40:43.755400 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.755366    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-888jp"]
Oct 30 02:40:43.759873 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:43.759836    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6386c9_d170_4b89_bb41_9a0641207188.slice/crio-5ff5daf0bd925a2b6cd7e5be0e05585e1918f345b1ee300049371d9b2ce983ed WatchSource:0}: Error finding container 5ff5daf0bd925a2b6cd7e5be0e05585e1918f345b1ee300049371d9b2ce983ed: Status 404 returned error can't find the container with id 5ff5daf0bd925a2b6cd7e5be0e05585e1918f345b1ee300049371d9b2ce983ed
Oct 30 02:40:43.794455 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.794431    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"]
Oct 30 02:40:43.798059 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:43.798036    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ea065c_0b57_43ec_99be_989bb703b2c5.slice/crio-8deb66594f98df1f78e7d9d7080e4829b8fc21c2fab810da82082ba96b13712d WatchSource:0}: Error finding container 8deb66594f98df1f78e7d9d7080e4829b8fc21c2fab810da82082ba96b13712d: Status 404 returned error can't find the container with id 8deb66594f98df1f78e7d9d7080e4829b8fc21c2fab810da82082ba96b13712d
Oct 30 02:40:43.801905 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.801886    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw"]
Oct 30 02:40:43.805807 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:43.805786    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2967919e_d201_48ae_b388_85fe023e1d21.slice/crio-c4d0066ee819fc8ce9c467b32fec17f97b20fc6fa6bc680a799b4613c5a342a1 WatchSource:0}: Error finding container c4d0066ee819fc8ce9c467b32fec17f97b20fc6fa6bc680a799b4613c5a342a1: Status 404 returned error can't find the container with id c4d0066ee819fc8ce9c467b32fec17f97b20fc6fa6bc680a799b4613c5a342a1
Oct 30 02:40:43.821066 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.821046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:43.821210 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.821193    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:40:43.821292 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.821245    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls podName:215f867e-a2ba-4b0d-ada3-6182c5e1a8fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.821231792 +0000 UTC m=+32.221803343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pppd8" (UID: "215f867e-a2ba-4b0d-ada3-6182c5e1a8fd") : secret "samples-operator-tls" not found
Oct 30 02:40:43.822640 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.822620    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j86rx"]
Oct 30 02:40:43.825618 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:43.825597    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea0be63_7705_40e8_b2a1_e48c39259c8b.slice/crio-6512af8f426058ddde7c7d59e9840651fffe4f32eb84ed2e5b6980b4ebaa07cb WatchSource:0}: Error finding container 6512af8f426058ddde7c7d59e9840651fffe4f32eb84ed2e5b6980b4ebaa07cb: Status 404 returned error can't find the container with id 6512af8f426058ddde7c7d59e9840651fffe4f32eb84ed2e5b6980b4ebaa07cb
Oct 30 02:40:43.862361 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.862331    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zh9rv"]
Oct 30 02:40:43.866179 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:43.866144    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76243024_183c_42e6_89fd_ccd82d3571c4.slice/crio-4c7cfaa83d792ca7515ddb62f7e21d968fa1fe46f5710a2f1d85f5c4f7c18190 WatchSource:0}: Error finding container 4c7cfaa83d792ca7515ddb62f7e21d968fa1fe46f5710a2f1d85f5c4f7c18190: Status 404 returned error can't find the container with id 4c7cfaa83d792ca7515ddb62f7e21d968fa1fe46f5710a2f1d85f5c4f7c18190
Oct 30 02:40:43.906593 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.906531    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9"]
Oct 30 02:40:43.910729 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:43.910678    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51189dec_b0c6_430e_9768_8f9532bdb22a.slice/crio-832e66d95d736de2ef3040fe215a29f0111e963aafff9582eb57a504793ccf4a WatchSource:0}: Error finding container 832e66d95d736de2ef3040fe215a29f0111e963aafff9582eb57a504793ccf4a: Status 404 returned error can't find the container with id 832e66d95d736de2ef3040fe215a29f0111e963aafff9582eb57a504793ccf4a
Oct 30 02:40:43.921686 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.921666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:43.921767 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.921701    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:43.921827 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.921813    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:40:43.921898 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.921839    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:40:43.921898 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.921867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:43.921898 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.921878    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert podName:69caed51-464d-4dab-b4a4-b4ff77da5e4e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.921864949 +0000 UTC m=+32.322436514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sj78j" (UID: "69caed51-464d-4dab-b4a4-b4ff77da5e4e") : secret "networking-console-plugin-cert" not found
Oct 30 02:40:43.922127 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.921927    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:40:43.922127 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.921980    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67544b9897-hb2j2: secret "image-registry-tls" not found
Oct 30 02:40:43.922127 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.921878    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56b7889b4-hzhbm: secret "image-registry-tls" not found
Oct 30 02:40:43.922127 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.922007    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls podName:cd57c8d3-5a2c-4068-b04a-901e5795b815 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.921998789 +0000 UTC m=+32.322570354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls") pod "image-registry-67544b9897-hb2j2" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815") : secret "image-registry-tls" not found
Oct 30 02:40:43.922127 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.922059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:43.922127 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.922119    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:43.922354 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.922131    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls podName:4f8a405e-f80a-4aa9-ab4c-2217e9250b70 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.922116495 +0000 UTC m=+32.322688051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls") pod "image-registry-56b7889b4-hzhbm" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70") : secret "image-registry-tls" not found
Oct 30 02:40:43.922354 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:43.922150    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls podName:16395f11-a919-4dd3-9d21-3388d472e616 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.922140767 +0000 UTC m=+32.322712319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls") pod "dns-default-m9rxl" (UID: "16395f11-a919-4dd3-9d21-3388d472e616") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:43.930034 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.930012    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7btgh"]
Oct 30 02:40:43.935731 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:43.935711    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4844aa91_37e4_4cae_a380_af0000c768fc.slice/crio-1b32445743e6f9c1ada4e2c374ae9fd6e4b3d9bce672b005023229590ec0e6c4 WatchSource:0}: Error finding container 1b32445743e6f9c1ada4e2c374ae9fd6e4b3d9bce672b005023229590ec0e6c4: Status 404 returned error can't find the container with id 1b32445743e6f9c1ada4e2c374ae9fd6e4b3d9bce672b005023229590ec0e6c4
Oct 30 02:40:43.956528 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:43.956499    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9r4jn"]
Oct 30 02:40:43.961356 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:43.961279    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0334aa2_bc83_440a_8781_60817828e1ab.slice/crio-82f612c6396937120581a6652be066f2edb8438f8e83271e6adc4bdee9ecea60 WatchSource:0}: Error finding container 82f612c6396937120581a6652be066f2edb8438f8e83271e6adc4bdee9ecea60: Status 404 returned error can't find the container with id 82f612c6396937120581a6652be066f2edb8438f8e83271e6adc4bdee9ecea60
Oct 30 02:40:44.345742 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.345655    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9r4jn" event={"ID":"e0334aa2-bc83-440a-8781-60817828e1ab","Type":"ContainerStarted","Data":"82f612c6396937120581a6652be066f2edb8438f8e83271e6adc4bdee9ecea60"}
Oct 30 02:40:44.347573 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.347541    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7btgh" event={"ID":"4844aa91-37e4-4cae-a380-af0000c768fc","Type":"ContainerStarted","Data":"d5ec8fa80a0040ab531239094f8a3bb479cc1deed1f309485b308500bda0210a"}
Oct 30 02:40:44.347693 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.347591    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7btgh" event={"ID":"4844aa91-37e4-4cae-a380-af0000c768fc","Type":"ContainerStarted","Data":"1b32445743e6f9c1ada4e2c374ae9fd6e4b3d9bce672b005023229590ec0e6c4"}
Oct 30 02:40:44.353389 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.353324    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9" event={"ID":"51189dec-b0c6-430e-9768-8f9532bdb22a","Type":"ContainerStarted","Data":"832e66d95d736de2ef3040fe215a29f0111e963aafff9582eb57a504793ccf4a"}
Oct 30 02:40:44.354733 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.354708    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp" event={"ID":"7b6386c9-d170-4b89-bb41-9a0641207188","Type":"ContainerStarted","Data":"5ff5daf0bd925a2b6cd7e5be0e05585e1918f345b1ee300049371d9b2ce983ed"}
Oct 30 02:40:44.355973 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.355907    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw" event={"ID":"2967919e-d201-48ae-b388-85fe023e1d21","Type":"ContainerStarted","Data":"c4d0066ee819fc8ce9c467b32fec17f97b20fc6fa6bc680a799b4613c5a342a1"}
Oct 30 02:40:44.357210 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.357153    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2" event={"ID":"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8","Type":"ContainerStarted","Data":"b44207e9edbdf6ff9dab11296e0e8b14384e4a82771a47746ce3624c2991ee4a"}
Oct 30 02:40:44.359129 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.359104    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zh9rv" event={"ID":"76243024-183c-42e6-89fd-ccd82d3571c4","Type":"ContainerStarted","Data":"4c7cfaa83d792ca7515ddb62f7e21d968fa1fe46f5710a2f1d85f5c4f7c18190"}
Oct 30 02:40:44.360493 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.360470    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w" event={"ID":"04ea065c-0b57-43ec-99be-989bb703b2c5","Type":"ContainerStarted","Data":"8deb66594f98df1f78e7d9d7080e4829b8fc21c2fab810da82082ba96b13712d"}
Oct 30 02:40:44.362527 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.362500    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cvtzh" event={"ID":"bf8a2708-a7fe-41b0-8f63-7850df383c8f","Type":"ContainerStarted","Data":"2519183f3012194decad5b242e07ec6841fec12988f92d4373169a5a7b75a670"}
Oct 30 02:40:44.362623 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.362543    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cvtzh" event={"ID":"bf8a2708-a7fe-41b0-8f63-7850df383c8f","Type":"ContainerStarted","Data":"72dd658f9795830c9a88e19578bdde58deb6ee4df059e5b8b0d8a1b205b11205"}
Oct 30 02:40:44.367768 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.367329    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j86rx" event={"ID":"1ea0be63-7705-40e8-b2a1-e48c39259c8b","Type":"ContainerStarted","Data":"73fdb52714897afb19b4b846a38651032f773a2467f5edf832f469d0823f75de"}
Oct 30 02:40:44.367768 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.367370    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j86rx" event={"ID":"1ea0be63-7705-40e8-b2a1-e48c39259c8b","Type":"ContainerStarted","Data":"6512af8f426058ddde7c7d59e9840651fffe4f32eb84ed2e5b6980b4ebaa07cb"}
Oct 30 02:40:44.387346 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.387294    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7btgh" podStartSLOduration=532.387277296 podStartE2EDuration="8m52.387277296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:44.387105725 +0000 UTC m=+31.787677327" watchObservedRunningTime="2026-10-30 02:40:44.387277296 +0000 UTC m=+31.787848866"
Oct 30 02:40:44.426283 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.426227    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cvtzh" podStartSLOduration=532.426207861 podStartE2EDuration="8m52.426207861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:44.424397809 +0000 UTC m=+31.824969383" watchObservedRunningTime="2026-10-30 02:40:44.426207861 +0000 UTC m=+31.826779436"
Oct 30 02:40:44.464774 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.463937    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-j86rx" podStartSLOduration=532.463922148 podStartE2EDuration="8m52.463922148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:44.463679366 +0000 UTC m=+31.864250939" watchObservedRunningTime="2026-10-30 02:40:44.463922148 +0000 UTC m=+31.864493724"
Oct 30 02:40:44.732630 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.732547    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:44.732801 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.732633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:44.732882 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.732831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:44.733067 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.733040    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.73300611 +0000 UTC m=+34.133577661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : configmap references non-existent config key: service-ca.crt
Oct 30 02:40:44.733169 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.733072    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:44.733169 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.733135    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls podName:c9aa32ac-0748-4d2f-820d-8a15ec78cdb1 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.733117548 +0000 UTC m=+34.133689119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gl8sd" (UID: "c9aa32ac-0748-4d2f-820d-8a15ec78cdb1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:44.733284 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.733210    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:40:44.733284 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.733245    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.733234422 +0000 UTC m=+34.133805988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : secret "router-metrics-certs-default" not found
Oct 30 02:40:44.834027 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.833993    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:44.834184 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.834148    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:40:44.834241 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.834218    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls podName:215f867e-a2ba-4b0d-ada3-6182c5e1a8fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.834201131 +0000 UTC m=+34.234772705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pppd8" (UID: "215f867e-a2ba-4b0d-ada3-6182c5e1a8fd") : secret "samples-operator-tls" not found
Oct 30 02:40:44.903768 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.900453    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.935292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.935362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.935402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:44.935469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.935627    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.935642    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67544b9897-hb2j2: secret "image-registry-tls" not found
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.935697    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls podName:cd57c8d3-5a2c-4068-b04a-901e5795b815 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.935679072 +0000 UTC m=+34.336250629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls") pod "image-registry-67544b9897-hb2j2" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815") : secret "image-registry-tls" not found
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.936106    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.936155    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls podName:16395f11-a919-4dd3-9d21-3388d472e616 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.936140781 +0000 UTC m=+34.336712335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls") pod "dns-default-m9rxl" (UID: "16395f11-a919-4dd3-9d21-3388d472e616") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.936218    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.936229    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56b7889b4-hzhbm: secret "image-registry-tls" not found
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.936260    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls podName:4f8a405e-f80a-4aa9-ab4c-2217e9250b70 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.936249334 +0000 UTC m=+34.336820891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls") pod "image-registry-56b7889b4-hzhbm" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70") : secret "image-registry-tls" not found
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.936311    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:40:44.936376 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:44.936343    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert podName:69caed51-464d-4dab-b4a4-b4ff77da5e4e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.936332936 +0000 UTC m=+34.336904491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sj78j" (UID: "69caed51-464d-4dab-b4a4-b4ff77da5e4e") : secret "networking-console-plugin-cert" not found
Oct 30 02:40:45.047667 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:45.047634    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:45.216227 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:45.216163    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:46.384556 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.384505    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw" event={"ID":"2967919e-d201-48ae-b388-85fe023e1d21","Type":"ContainerStarted","Data":"26fc8021e7a47da8b1f36933e3c9002a36da7bcaa2ee9e15814cb7523929bb15"}
Oct 30 02:40:46.424269 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.424209    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw" podStartSLOduration=508.175051583 podStartE2EDuration="8m30.424192027s" totalImagesPullingTime="2.249140444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:46.423205963 +0000 UTC m=+33.823777565" watchObservedRunningTime="2026-10-30 02:40:46.424192027 +0000 UTC m=+33.824763600"
Oct 30 02:40:46.548015 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.547951    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:46.670414 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.670325    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:46.757384 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.757347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:46.758398 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.757944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:46.758398 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.758021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:46.758398 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.758182    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:40:46.758398 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.758234    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.75821648 +0000 UTC m=+38.158788047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : secret "router-metrics-certs-default" not found
Oct 30 02:40:46.758398 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.758314    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:46.758398 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.758348    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls podName:c9aa32ac-0748-4d2f-820d-8a15ec78cdb1 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.758337178 +0000 UTC m=+38.158908742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gl8sd" (UID: "c9aa32ac-0748-4d2f-820d-8a15ec78cdb1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:46.758398 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.758367    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.758357744 +0000 UTC m=+38.158929300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : configmap references non-existent config key: service-ca.crt
Oct 30 02:40:46.859593 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.859027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:46.859593 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.859091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:46.859593 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.859166    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:40:46.859593 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.859253    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:40:46.859593 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.859254    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls podName:215f867e-a2ba-4b0d-ada3-6182c5e1a8fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.859230249 +0000 UTC m=+38.259801822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pppd8" (UID: "215f867e-a2ba-4b0d-ada3-6182c5e1a8fd") : secret "samples-operator-tls" not found
Oct 30 02:40:46.859593 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.859323    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs podName:2c04bf2f-3e60-442a-8639-10feae0a3f96 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:18.859307302 +0000 UTC m=+66.259878859 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs") pod "network-metrics-daemon-pt72w" (UID: "2c04bf2f-3e60-442a-8639-10feae0a3f96") : secret "metrics-daemon-secret" not found
Oct 30 02:40:46.961159 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.960204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrsj\" (UniqueName: \"kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj\") pod \"network-check-target-tb8xt\" (UID: \"00bc8689-1f9b-40a7-9f19-972395d1bc69\") " pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:46.961159 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.960271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:46.961159 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.960327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:46.961159 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.960365    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:46.961159 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.960407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:46.961159 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.960445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:46.961159 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.960608    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:40:46.961159 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.960624    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67544b9897-hb2j2: secret "image-registry-tls" not found
Oct 30 02:40:46.961159 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.960682    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls podName:cd57c8d3-5a2c-4068-b04a-901e5795b815 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.960663692 +0000 UTC m=+38.361235247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls") pod "image-registry-67544b9897-hb2j2" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815") : secret "image-registry-tls" not found
Oct 30 02:40:46.962089 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.961810    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:40:46.962089 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.961832    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56b7889b4-hzhbm: secret "image-registry-tls" not found
Oct 30 02:40:46.962089 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.961896    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls podName:4f8a405e-f80a-4aa9-ab4c-2217e9250b70 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.961879466 +0000 UTC m=+38.362451026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls") pod "image-registry-56b7889b4-hzhbm" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70") : secret "image-registry-tls" not found
Oct 30 02:40:46.962089 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.961957    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:40:46.962089 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.961994    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert podName:69caed51-464d-4dab-b4a4-b4ff77da5e4e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.961982397 +0000 UTC m=+38.362553955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sj78j" (UID: "69caed51-464d-4dab-b4a4-b4ff77da5e4e") : secret "networking-console-plugin-cert" not found
Oct 30 02:40:46.962089 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.961807    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:46.962089 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:46.962030    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls podName:16395f11-a919-4dd3-9d21-3388d472e616 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.962019752 +0000 UTC m=+38.362591303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls") pod "dns-default-m9rxl" (UID: "16395f11-a919-4dd3-9d21-3388d472e616") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:46.965934 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.965908    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:46.970318 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:46.970296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzrsj\" (UniqueName: \"kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj\") pod \"network-check-target-tb8xt\" (UID: \"00bc8689-1f9b-40a7-9f19-972395d1bc69\") " pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:47.049234 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:47.049184    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:47.126133 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:47.125922    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:47.383062 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:47.382990    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tb8xt"]
Oct 30 02:40:47.394737 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:47.394711    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bc8689_1f9b_40a7_9f19_972395d1bc69.slice/crio-21df96b6ed9f5503f4eff27cc64c39595412bd7860c5cb28f467edc77ffafa9d WatchSource:0}: Error finding container 21df96b6ed9f5503f4eff27cc64c39595412bd7860c5cb28f467edc77ffafa9d: Status 404 returned error can't find the container with id 21df96b6ed9f5503f4eff27cc64c39595412bd7860c5cb28f467edc77ffafa9d
Oct 30 02:40:47.466603 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:47.466075    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hzqcm"]
Oct 30 02:40:47.473377 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:47.473317    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bf413f_45b3_450c_b797_4e674e8da8ec.slice/crio-f5044505814979be024a50e25da60cb9036e68c8b6bb7cec66a5efa5053bf6eb WatchSource:0}: Error finding container f5044505814979be024a50e25da60cb9036e68c8b6bb7cec66a5efa5053bf6eb: Status 404 returned error can't find the container with id f5044505814979be024a50e25da60cb9036e68c8b6bb7cec66a5efa5053bf6eb
Oct 30 02:40:47.595031 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:47.594998    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:48.392767 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:48.392715    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hzqcm" event={"ID":"a2bf413f-45b3-450c-b797-4e674e8da8ec","Type":"ContainerStarted","Data":"f5044505814979be024a50e25da60cb9036e68c8b6bb7cec66a5efa5053bf6eb"}
Oct 30 02:40:48.394303 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:48.394264    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tb8xt" event={"ID":"00bc8689-1f9b-40a7-9f19-972395d1bc69","Type":"ContainerStarted","Data":"21df96b6ed9f5503f4eff27cc64c39595412bd7860c5cb28f467edc77ffafa9d"}
Oct 30 02:40:49.405309 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.405277    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9r4jn" event={"ID":"e0334aa2-bc83-440a-8781-60817828e1ab","Type":"ContainerStarted","Data":"594557d73811e5ec3865ac8b7cf0cc582492ddb037f47176b14364211e9f9a46"}
Oct 30 02:40:49.409810 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.409786    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/0.log"
Oct 30 02:40:49.410718 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.410024    2576 generic.go:356] "Generic (PLEG): container finished" podID="7b6386c9-d170-4b89-bb41-9a0641207188" containerID="b779a3cd9a8c512be4f0e72c2da5b8571cd157d555eb61077d117f9274a946d5" exitCode=255
Oct 30 02:40:49.410718 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.410126    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp" event={"ID":"7b6386c9-d170-4b89-bb41-9a0641207188","Type":"ContainerDied","Data":"b779a3cd9a8c512be4f0e72c2da5b8571cd157d555eb61077d117f9274a946d5"}
Oct 30 02:40:49.410718 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.410420    2576 scope.go:122] "RemoveContainer" containerID="b779a3cd9a8c512be4f0e72c2da5b8571cd157d555eb61077d117f9274a946d5"
Oct 30 02:40:49.423630 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.423271    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2" event={"ID":"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8","Type":"ContainerStarted","Data":"31b7a01f469f78fcf5169fbc266f77b0a2d489383c5edbfce77fdf772b26cffb"}
Oct 30 02:40:49.432863 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.432383    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zh9rv" event={"ID":"76243024-183c-42e6-89fd-ccd82d3571c4","Type":"ContainerStarted","Data":"e758dd753f771f94e9471c3230c808507ea3d3b1d7cca0eabfb5fab77efcec2a"}
Oct 30 02:40:49.435000 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.434976    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w" event={"ID":"04ea065c-0b57-43ec-99be-989bb703b2c5","Type":"ContainerStarted","Data":"5f11d73b2bd4ead981af1cc4d6c8e267a126fb29c3fe412efc08c9188ce1ad54"}
Oct 30 02:40:49.447143 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.446917    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9r4jn" podStartSLOduration=2.360867009 podStartE2EDuration="7.446901458s" totalImagesPullingTime="5.086034449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:49.446284194 +0000 UTC m=+36.846855772" watchObservedRunningTime="2026-10-30 02:40:49.446901458 +0000 UTC m=+36.847473036"
Oct 30 02:40:49.487863 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.487798    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2" podStartSLOduration=508.760751517 podStartE2EDuration="8m33.487780152s" totalImagesPullingTime="4.727028635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:49.487266141 +0000 UTC m=+36.887837717" watchObservedRunningTime="2026-10-30 02:40:49.487780152 +0000 UTC m=+36.888351729"
Oct 30 02:40:49.520635 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.515873    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kkfq8"]
Oct 30 02:40:49.535623 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.534723    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kkfq8"]
Oct 30 02:40:49.535623 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.534928    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.541267 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.539599    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w" podStartSLOduration=508.297178671 podStartE2EDuration="8m33.539583821s" totalImagesPullingTime="5.24240515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:49.538983985 +0000 UTC m=+36.939555559" watchObservedRunningTime="2026-10-30 02:40:49.539583821 +0000 UTC m=+36.940155394"
Oct 30 02:40:49.552317 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.551299    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:40:49.558876 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.554664    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:40:49.558876 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.554801    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rd75s\""
Oct 30 02:40:49.609885 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.609841    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-q8xpw-hdnw7"]
Oct 30 02:40:49.611832 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.611595    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-crio-socket\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.627693 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.627667    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.637356 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.637329    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q8xpw-hdnw7"]
Oct 30 02:40:49.649315 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.648355    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-zh9rv" podStartSLOduration=509.041613883 podStartE2EDuration="8m33.64833996s" totalImagesPullingTime="4.606726077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:49.648019034 +0000 UTC m=+37.048590609" watchObservedRunningTime="2026-10-30 02:40:49.64833996 +0000 UTC m=+37.048911533"
Oct 30 02:40:49.651135 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.651111    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:49.656352 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.656299    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mppsd\""
Oct 30 02:40:49.716506 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.716473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.716648 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.716591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-crio-socket\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.716648 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.716640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55nbl\" (UniqueName: \"kubernetes.io/projected/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-api-access-55nbl\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.716767 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.716750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-data-volume\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.716817 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.716786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.717111 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.717081    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-crio-socket\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.815571 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.815528    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:49.817585 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.817553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-data-volume\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.817720 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.817595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.817720 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.817642    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-trusted-ca-bundle\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.817800 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:49.817778    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:49.817875 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:49.817840    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.317823491 +0000 UTC m=+37.718395055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:49.817996 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.817911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.817996 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.817950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d786d081-b91d-475a-937b-021d17c210b1-archives-path\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.817996 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.817979    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-data-volume\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.818161 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.818020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-service-ca-bundle\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.818161 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:49.818045    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:40:49.818161 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:49.818094    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.31807737 +0000 UTC m=+37.718648932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : configmap "kube-rbac-proxy" not found
Oct 30 02:40:49.818161 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.818115    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5wg8\" (UniqueName: \"kubernetes.io/projected/d786d081-b91d-475a-937b-021d17c210b1-kube-api-access-b5wg8\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.818379 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.818226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55nbl\" (UniqueName: \"kubernetes.io/projected/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-api-access-55nbl\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.844225 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.844189    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55nbl\" (UniqueName: \"kubernetes.io/projected/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-api-access-55nbl\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:49.851211 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.851003    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5"]
Oct 30 02:40:49.872469 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.872450    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5"]
Oct 30 02:40:49.873057 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.872693    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5"
Oct 30 02:40:49.888579 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.888558    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5"]
Oct 30 02:40:49.888673 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.888657    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5"
Oct 30 02:40:49.900675 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.900651    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5"]
Oct 30 02:40:49.919163 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.919092    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5wg8\" (UniqueName: \"kubernetes.io/projected/d786d081-b91d-475a-937b-021d17c210b1-kube-api-access-b5wg8\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.919268 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.919157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw4rt\" (UniqueName: \"kubernetes.io/projected/7f9c5ebb-f865-483c-9ee3-a44c74aa3abe-kube-api-access-kw4rt\") pod \"migrator-66df5d496b-vd9k5\" (UID: \"7f9c5ebb-f865-483c-9ee3-a44c74aa3abe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5"
Oct 30 02:40:49.919365 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.919346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-trusted-ca-bundle\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.919435 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.919418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d786d081-b91d-475a-937b-021d17c210b1-archives-path\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.919486 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.919463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-service-ca-bundle\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.920037 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.920011    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-service-ca-bundle\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.920313 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.920293    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d786d081-b91d-475a-937b-021d17c210b1-archives-path\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.920401 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.920350    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-trusted-ca-bundle\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.924962 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.924945    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xh5tx\""
Oct 30 02:40:49.938831 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.938810    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5wg8\" (UniqueName: \"kubernetes.io/projected/d786d081-b91d-475a-937b-021d17c210b1-kube-api-access-b5wg8\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:49.949661 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.949639    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:49.962830 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.962813    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:40:49.962830 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:49.962813    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:50.004963 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.004940    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:50.020441 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.020412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fzsx\" (UniqueName: \"kubernetes.io/projected/00b0139b-56b5-4daa-8fef-b86d0ac5a1d8-kube-api-access-4fzsx\") pod \"migrator-66df5d496b-8slr5\" (UID: \"00b0139b-56b5-4daa-8fef-b86d0ac5a1d8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5"
Oct 30 02:40:50.020537 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.020502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw4rt\" (UniqueName: \"kubernetes.io/projected/7f9c5ebb-f865-483c-9ee3-a44c74aa3abe-kube-api-access-kw4rt\") pod \"migrator-66df5d496b-vd9k5\" (UID: \"7f9c5ebb-f865-483c-9ee3-a44c74aa3abe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5"
Oct 30 02:40:50.044032 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.044006    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw4rt\" (UniqueName: \"kubernetes.io/projected/7f9c5ebb-f865-483c-9ee3-a44c74aa3abe-kube-api-access-kw4rt\") pod \"migrator-66df5d496b-vd9k5\" (UID: \"7f9c5ebb-f865-483c-9ee3-a44c74aa3abe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5"
Oct 30 02:40:50.080353 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.079946    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:50.121686 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.121650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzsx\" (UniqueName: \"kubernetes.io/projected/00b0139b-56b5-4daa-8fef-b86d0ac5a1d8-kube-api-access-4fzsx\") pod \"migrator-66df5d496b-8slr5\" (UID: \"00b0139b-56b5-4daa-8fef-b86d0ac5a1d8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5"
Oct 30 02:40:50.141409 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.141384    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzsx\" (UniqueName: \"kubernetes.io/projected/00b0139b-56b5-4daa-8fef-b86d0ac5a1d8-kube-api-access-4fzsx\") pod \"migrator-66df5d496b-8slr5\" (UID: \"00b0139b-56b5-4daa-8fef-b86d0ac5a1d8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5"
Oct 30 02:40:50.201876 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.201778    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5"
Oct 30 02:40:50.206175 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.206140    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:50.218623 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.218597    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5"
Oct 30 02:40:50.323984 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.323953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:50.324084 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.324039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:50.324257 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.324198    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:40:50.324257 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.324248    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:51.324235186 +0000 UTC m=+38.724806750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : configmap "kube-rbac-proxy" not found
Oct 30 02:40:50.324580 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.324567    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:50.324616 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.324607    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:51.324596905 +0000 UTC m=+38.725168469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:50.327925 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.327897    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q8xpw-hdnw7"]
Oct 30 02:40:50.440468 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.440437    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9" event={"ID":"51189dec-b0c6-430e-9768-8f9532bdb22a","Type":"ContainerStarted","Data":"24465850e0240f031de9f7ebe74a2e44aa7f15cc7e27769ffecc71f13c08518d"}
Oct 30 02:40:50.442944 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.442923    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:40:50.444049 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.443394    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/0.log"
Oct 30 02:40:50.444049 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.443432    2576 generic.go:356] "Generic (PLEG): container finished" podID="7b6386c9-d170-4b89-bb41-9a0641207188" containerID="b964cb6f1289891b21a2f16154013500efe4d5530fd822329a568441f26a926f" exitCode=255
Oct 30 02:40:50.444049 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.443710    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp" event={"ID":"7b6386c9-d170-4b89-bb41-9a0641207188","Type":"ContainerDied","Data":"b964cb6f1289891b21a2f16154013500efe4d5530fd822329a568441f26a926f"}
Oct 30 02:40:50.444049 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.443763    2576 scope.go:122] "RemoveContainer" containerID="b779a3cd9a8c512be4f0e72c2da5b8571cd157d555eb61077d117f9274a946d5"
Oct 30 02:40:50.444049 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.443786    2576 scope.go:122] "RemoveContainer" containerID="b964cb6f1289891b21a2f16154013500efe4d5530fd822329a568441f26a926f"
Oct 30 02:40:50.444049 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.443972    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-888jp_openshift-console-operator(7b6386c9-d170-4b89-bb41-9a0641207188)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp" podUID="7b6386c9-d170-4b89-bb41-9a0641207188"
Oct 30 02:40:50.446314 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.445539    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8xpw-hdnw7" event={"ID":"d786d081-b91d-475a-937b-021d17c210b1","Type":"ContainerStarted","Data":"9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659"}
Oct 30 02:40:50.446314 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.445576    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8xpw-hdnw7" event={"ID":"d786d081-b91d-475a-937b-021d17c210b1","Type":"ContainerStarted","Data":"c161763cc1a81d0ec45280c251660d734c6ee1894a175d8969c2ab828452effa"}
Oct 30 02:40:50.519448 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.519401    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9" podStartSLOduration=504.028880507 podStartE2EDuration="8m29.519381204s" totalImagesPullingTime="5.490500697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:50.484388607 +0000 UTC m=+37.884960181" watchObservedRunningTime="2026-10-30 02:40:50.519381204 +0000 UTC m=+37.919952776"
Oct 30 02:40:50.521144 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.521124    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5"]
Oct 30 02:40:50.525880 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:50.525765    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b0139b_56b5_4daa_8fef_b86d0ac5a1d8.slice/crio-fe83b4c8f25d473ad47fc26249f164956aeb8979ce91f591f3d63d9778913bbe WatchSource:0}: Error finding container fe83b4c8f25d473ad47fc26249f164956aeb8979ce91f591f3d63d9778913bbe: Status 404 returned error can't find the container with id fe83b4c8f25d473ad47fc26249f164956aeb8979ce91f591f3d63d9778913bbe
Oct 30 02:40:50.548765 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.548744    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5"]
Oct 30 02:40:50.552376 ip-10-0-136-238 kubenswrapper[2576]: W1030 02:40:50.552355    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9c5ebb_f865_483c_9ee3_a44c74aa3abe.slice/crio-71c7d22f628a11906ecbc82dceded6e9ec5c3b8f9b35f164eb4c0587129a7b87 WatchSource:0}: Error finding container 71c7d22f628a11906ecbc82dceded6e9ec5c3b8f9b35f164eb4c0587129a7b87: Status 404 returned error can't find the container with id 71c7d22f628a11906ecbc82dceded6e9ec5c3b8f9b35f164eb4c0587129a7b87
Oct 30 02:40:50.565710 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.565673    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-q8xpw-hdnw7" podStartSLOduration=1.565657836 podStartE2EDuration="1.565657836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:50.56427167 +0000 UTC m=+37.964843245" watchObservedRunningTime="2026-10-30 02:40:50.565657836 +0000 UTC m=+37.966229400"
Oct 30 02:40:50.652080 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.652044    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8t2p7"]
Oct 30 02:40:50.677976 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.677950    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8t2p7"]
Oct 30 02:40:50.678093 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.678037    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:50.693079 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.693054    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 02:40:50.693181 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.693121    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 02:40:50.693181 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.693155    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 02:40:50.693181 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.693157    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 02:40:50.693181 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.693164    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vgdcn\""
Oct 30 02:40:50.728106 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.728082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-cabundle\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:50.728255 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.728237    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrc7\" (UniqueName: \"kubernetes.io/projected/7079cfbc-daff-4782-b3ca-e5718d1d1421-kube-api-access-6qrc7\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:50.728385 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.728363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-key\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:50.829000 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.828970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-cabundle\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:50.829169 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.829052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:50.829239 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.829209    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:58.829191328 +0000 UTC m=+46.229762893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : configmap references non-existent config key: service-ca.crt
Oct 30 02:40:50.829309 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.829240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrc7\" (UniqueName: \"kubernetes.io/projected/7079cfbc-daff-4782-b3ca-e5718d1d1421-kube-api-access-6qrc7\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:50.829309 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.829295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:50.829407 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.829347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:50.829407 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.829381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-key\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:50.829905 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.829616    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:50.829905 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.829679    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls podName:c9aa32ac-0748-4d2f-820d-8a15ec78cdb1 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:58.829665136 +0000 UTC m=+46.230236687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gl8sd" (UID: "c9aa32ac-0748-4d2f-820d-8a15ec78cdb1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:50.829905 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.829625    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:40:50.829905 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.829727    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:58.829718 +0000 UTC m=+46.230289552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : secret "router-metrics-certs-default" not found
Oct 30 02:40:50.829905 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.829739    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-cabundle\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:50.832021 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.832002    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-key\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:50.850666 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.850645    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrc7\" (UniqueName: \"kubernetes.io/projected/7079cfbc-daff-4782-b3ca-e5718d1d1421-kube-api-access-6qrc7\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:50.930690 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:50.930629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:50.930878 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.930761    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 02:40:50.930878 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:50.930842    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls podName:215f867e-a2ba-4b0d-ada3-6182c5e1a8fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:58.930820701 +0000 UTC m=+46.331392273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pppd8" (UID: "215f867e-a2ba-4b0d-ada3-6182c5e1a8fd") : secret "samples-operator-tls" not found
Oct 30 02:40:51.003735 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.003095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:51.031295 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.031263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:51.031460 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.031393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:51.031460 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.031452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:51.032071 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.031502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:51.032071 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.031639    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 02:40:51.032071 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.031703    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert podName:69caed51-464d-4dab-b4a4-b4ff77da5e4e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:59.031684911 +0000 UTC m=+46.432256464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sj78j" (UID: "69caed51-464d-4dab-b4a4-b4ff77da5e4e") : secret "networking-console-plugin-cert" not found
Oct 30 02:40:51.032327 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.032108    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:40:51.032327 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.032125    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67544b9897-hb2j2: secret "image-registry-tls" not found
Oct 30 02:40:51.032327 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.032198    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls podName:cd57c8d3-5a2c-4068-b04a-901e5795b815 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:59.032181519 +0000 UTC m=+46.432753084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls") pod "image-registry-67544b9897-hb2j2" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815") : secret "image-registry-tls" not found
Oct 30 02:40:51.032327 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.032260    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:51.032327 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.032295    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls podName:16395f11-a919-4dd3-9d21-3388d472e616 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:59.03228329 +0000 UTC m=+46.432854846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls") pod "dns-default-m9rxl" (UID: "16395f11-a919-4dd3-9d21-3388d472e616") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:51.032631 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.032367    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 02:40:51.032631 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.032378    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56b7889b4-hzhbm: secret "image-registry-tls" not found
Oct 30 02:40:51.032631 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.032410    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls podName:4f8a405e-f80a-4aa9-ab4c-2217e9250b70 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:59.032398772 +0000 UTC m=+46.432970323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls") pod "image-registry-56b7889b4-hzhbm" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70") : secret "image-registry-tls" not found
Oct 30 02:40:51.171807 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.171755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:51.330112 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.330077    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8t2p7"]
Oct 30 02:40:51.331800 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.331649    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:51.335858 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.334766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:51.335858 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.334842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:51.335858 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.335152    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:40:51.335858 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.335167    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:51.335858 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.335212    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:53.335195708 +0000 UTC m=+40.735767274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : configmap "kube-rbac-proxy" not found
Oct 30 02:40:51.335858 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.335232    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:53.335222935 +0000 UTC m=+40.735794488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:51.453881 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.453828    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:40:51.454332 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.454265    2576 scope.go:122] "RemoveContainer" containerID="b964cb6f1289891b21a2f16154013500efe4d5530fd822329a568441f26a926f"
Oct 30 02:40:51.454528 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:51.454476    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-888jp_openshift-console-operator(7b6386c9-d170-4b89-bb41-9a0641207188)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp" podUID="7b6386c9-d170-4b89-bb41-9a0641207188"
Oct 30 02:40:51.458388 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.458363    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tb8xt" event={"ID":"00bc8689-1f9b-40a7-9f19-972395d1bc69","Type":"ContainerStarted","Data":"7dea2e49d7c1b68e6c5bc562347440bc01c0246f8933437e314c714730d04c8e"}
Oct 30 02:40:51.458886 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.458869    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:51.460128 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.460103    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5" event={"ID":"7f9c5ebb-f865-483c-9ee3-a44c74aa3abe","Type":"ContainerStarted","Data":"71c7d22f628a11906ecbc82dceded6e9ec5c3b8f9b35f164eb4c0587129a7b87"}
Oct 30 02:40:51.461758 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.461737    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7" event={"ID":"7079cfbc-daff-4782-b3ca-e5718d1d1421","Type":"ContainerStarted","Data":"c31eddff615841c494dd1fdd5e6fbcb28f31b024219676055d9b8d40c98d375d"}
Oct 30 02:40:51.461894 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.461776    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7" event={"ID":"7079cfbc-daff-4782-b3ca-e5718d1d1421","Type":"ContainerStarted","Data":"1209f1414c30f5e2ef9460b8f1a063f56a5d0e68f861c9e020a5027a359d52a5"}
Oct 30 02:40:51.463512 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.463494    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5" event={"ID":"00b0139b-56b5-4daa-8fef-b86d0ac5a1d8","Type":"ContainerStarted","Data":"fe83b4c8f25d473ad47fc26249f164956aeb8979ce91f591f3d63d9778913bbe"}
Oct 30 02:40:51.542199 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.542138    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7" podStartSLOduration=1.542119776 podStartE2EDuration="1.542119776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:51.541941573 +0000 UTC m=+38.942513150" watchObservedRunningTime="2026-10-30 02:40:51.542119776 +0000 UTC m=+38.942691342"
Oct 30 02:40:51.586911 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.586787    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tb8xt" podStartSLOduration=35.462892109 podStartE2EDuration="38.586769989s" totalImagesPullingTime="3.12387788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:51.58535047 +0000 UTC m=+38.985922045" watchObservedRunningTime="2026-10-30 02:40:51.586769989 +0000 UTC m=+38.987341565"
Oct 30 02:40:51.624140 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.624105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:51.862180 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.862105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:51.994996 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:51.994964    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:52.312912 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:52.310294    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:52.428583 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:52.428544    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:52.468874 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:52.468822    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5" event={"ID":"00b0139b-56b5-4daa-8fef-b86d0ac5a1d8","Type":"ContainerStarted","Data":"de71ce44640388a7acd120a46e6d30fb80f1c0aa703a7a35c30510192f472c0b"}
Oct 30 02:40:52.516784 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:52.516734    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5" podStartSLOduration=1.849410115 podStartE2EDuration="3.516713699s" totalImagesPullingTime="1.667303584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:52.516121343 +0000 UTC m=+39.916692917" watchObservedRunningTime="2026-10-30 02:40:52.516713699 +0000 UTC m=+39.917285272"
Oct 30 02:40:52.583673 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:52.583646    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:53.046445 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.046415    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qvbmn_5d7330ef-3e60-407f-89ca-0c0c9e38c87d/dns-node-resolver/0.log"
Oct 30 02:40:53.130662 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.130627    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:53.252396 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.252362    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:53.355357 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.355275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:53.355573 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.355372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:53.355629 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:53.355587    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:40:53.355688 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:53.355655    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:57.355633616 +0000 UTC m=+44.756205168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : configmap "kube-rbac-proxy" not found
Oct 30 02:40:53.355902 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:53.355879    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:53.356015 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:53.355948    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:57.355929723 +0000 UTC m=+44.756501274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:53.438557 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.438527    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:53.438727 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.438583    2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:53.438991 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.438977    2576 scope.go:122] "RemoveContainer" containerID="b964cb6f1289891b21a2f16154013500efe4d5530fd822329a568441f26a926f"
Oct 30 02:40:53.439197 ip-10-0-136-238 kubenswrapper[2576]: E1030 02:40:53.439171    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-888jp_openshift-console-operator(7b6386c9-d170-4b89-bb41-9a0641207188)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp" podUID="7b6386c9-d170-4b89-bb41-9a0641207188"
Oct 30 02:40:53.474439 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.474405    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5" event={"ID":"7f9c5ebb-f865-483c-9ee3-a44c74aa3abe","Type":"ContainerStarted","Data":"e1ef3aa9b38e55e2ebde54331d10a2c6e064672742b121c61c0985c6c1587f43"}
Oct 30 02:40:53.513619 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.513553    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5" podStartSLOduration=2.671046624 podStartE2EDuration="4.513535537s" totalImagesPullingTime="1.842488913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:53.511803224 +0000 UTC m=+40.912374797" watchObservedRunningTime="2026-10-30 02:40:53.513535537 +0000 UTC m=+40.914107111"
Oct 30 02:40:53.731409 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.731308    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:53.846946 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.846918    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hqwcj_af778722-0e75-4ddd-9819-d783cfa858e1/node-ca/0.log"
Oct 30 02:40:53.933776 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:53.933744    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:54.337124 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:54.337095    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:54.453837 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:54.453808    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:54.480149 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:54.480121    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hzqcm" event={"ID":"a2bf413f-45b3-450c-b797-4e674e8da8ec","Type":"ContainerStarted","Data":"8dd81504a4d5231ae52c97f63c6ac73c083604074e13ba8cd1ea4df266aa29e6"}
Oct 30 02:40:54.522724 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:54.522678    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hzqcm" podStartSLOduration=34.657531694 podStartE2EDuration="41.522662975s" totalImagesPullingTime="6.865131281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:54.522080109 +0000 UTC m=+41.922651687" watchObservedRunningTime="2026-10-30 02:40:54.522662975 +0000 UTC m=+41.923234581"
Oct 30 02:40:54.571285 ip-10-0-136-238 kubenswrapper[2576]: I1030 02:40:54.571253    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:54.641910 ip-10-0-136-238 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 02:40:54.656265 ip-10-0-136-238 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 02:40:54.656487 ip-10-0-136-238 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 02:40:54.656549 ip-10-0-136-238 systemd[1]: kubelet.service: Consumed 3.151s CPU time.
Oct 30 02:40:54.667466 ip-10-0-136-238 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:40:54.761543 ip-10-0-136-238 kubenswrapper[5061]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:54.761543 ip-10-0-136-238 kubenswrapper[5061]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:40:54.761543 ip-10-0-136-238 kubenswrapper[5061]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:54.761543 ip-10-0-136-238 kubenswrapper[5061]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:54.763634 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763553    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:54.763634 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763635    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763639    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763643    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763645    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763648    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763651    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763655    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763658    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763661    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763663    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763666    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763669    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763673    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763676    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763678    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763681    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763683    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763686    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763689    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:54.763705 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763691    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763694    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763696    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763699    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763702    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763704    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763707    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763710    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763713    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763718    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763722    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763725    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763728    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763731    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763733    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763736    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763739    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763741    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763744    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763746    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:54.764180 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763749    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763752    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763754    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763757    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763760    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763762    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763766    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763768    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763771    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763774    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763777    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763780    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763782    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763785    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763788    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763791    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763794    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763801    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763804    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763807    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:54.764670 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763809    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763812    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763815    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763817    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763820    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763822    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763826    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763830    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763834    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763837    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763840    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763857    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763862    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763866    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763870    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763873    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763876    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763879    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763881    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:54.765202 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763885    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763887    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763890    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763892    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763895    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763897    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763901    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763904    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763907    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763910    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763913    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763916    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763919    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763921    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763924    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763927    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763929    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763932    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763935    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:54.765657 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763937    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763940    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763942    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763945    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763947    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763950    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763953    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.763955    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764707    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764714    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764717    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764720    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764732    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764741    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764749    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764753    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764756    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764759    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764761    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:54.766143 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764764    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764767    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764770    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764772    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764775    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764778    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764781    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764784    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764787    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764789    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764792    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764795    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764797    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764800    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764802    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764805    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764808    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764810    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764813    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764817    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:54.766616 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764820    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764823    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764826    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764828    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764831    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764833    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764836    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764838    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764842    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764860    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764863    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764867    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764880    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764883    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764886    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764889    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764891    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764895    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764898    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764901    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:54.767126 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764903    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764906    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764908    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764911    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764914    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764916    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764919    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764922    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764924    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764927    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764929    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764932    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764935    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764939    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764943    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764946    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764949    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764952    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764954    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:54.767682 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764957    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764960    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764963    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764966    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764968    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764971    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764973    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764977    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764980    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764982    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764985    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764987    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764990    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764992    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764995    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.764997    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765000    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765003    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765005    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765008    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:54.768181 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765010    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765013    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765015    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765018    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765021    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765024    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765026    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765029    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765031    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765034    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765036    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765039    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765042    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765044    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765047    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765051    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765556    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765563    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765567    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765570    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:54.768674 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765572    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765575    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765578    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765581    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765585    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765589    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765593    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765595    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765598    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765601    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765604    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765606    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765609    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765611    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765614    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765617    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765619    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765622    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765625    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:54.769222 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765627    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765630    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765632    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765636    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765640    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765646    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765649    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765652    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765655    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765658    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765660    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765663    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765667    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765669    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765672    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765675    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765677    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765680    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765683    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:54.769692 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765685    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765688    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765690    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765693    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765695    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765698    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765700    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765703    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765706    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765708    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765711    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765713    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765716    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765718    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765721    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765724    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765726    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765729    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765733    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:54.770254 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765735    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765738    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765741    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765744    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765747    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765750    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765752    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765755    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765758    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765760    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765763    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765766    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765769    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765771    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765774    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765777    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765780    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765782    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765785    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765787    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:54.771055 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765790    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765792    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765795    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765797    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765800    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765802    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765805    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765807    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765810    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765812    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765815    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765819    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765822    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765824    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765827    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765830    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765833    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765836    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765839    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765842    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:54.771763 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765860    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765863    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765866    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765869    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.765871    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.765967    5061 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.765980    5061 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.765986    5061 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.765991    5061 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.765995    5061 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.765999    5061 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766003    5061 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766007    5061 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766011    5061 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766014    5061 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766017    5061 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766020    5061 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766023    5061 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766026    5061 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766029    5061 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766032    5061 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766035    5061 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766038    5061 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:40:54.772284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766044    5061 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766049    5061 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766052    5061 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766055    5061 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766059    5061 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766062    5061 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766066    5061 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766069    5061 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766073    5061 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766076    5061 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766079    5061 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766082    5061 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766085    5061 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766088    5061 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766092    5061 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766095    5061 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766099    5061 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766102    5061 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766105    5061 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766108    5061 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766112    5061 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766115    5061 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766119    5061 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766122    5061 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:40:54.772863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766125    5061 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766129    5061 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766132    5061 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766135    5061 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766139    5061 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766142    5061 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766145    5061 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766148    5061 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766152    5061 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766157    5061 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766160    5061 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766163    5061 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766167    5061 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766170    5061 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766173    5061 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766177    5061 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766180    5061 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766184    5061 flags.go:64] FLAG: --help="false"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766187    5061 flags.go:64] FLAG: --hostname-override="ip-10-0-136-238.ec2.internal"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766190    5061 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766193    5061 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766196    5061 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766199    5061 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:40:54.773504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766203    5061 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766206    5061 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766209    5061 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766212    5061 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766215    5061 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766217    5061 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766221    5061 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766224    5061 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766227    5061 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766230    5061 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766233    5061 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766236    5061 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766239    5061 flags.go:64] FLAG: --lock-file=""
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766242    5061 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766245    5061 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766248    5061 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766253    5061 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766257    5061 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766262    5061 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766265    5061 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766267    5061 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766271    5061 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766274    5061 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766277    5061 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766281    5061 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:40:54.774087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766284    5061 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766288    5061 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766292    5061 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766295    5061 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766298    5061 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766301    5061 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766304    5061 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766307    5061 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766310    5061 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766317    5061 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766320    5061 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766323    5061 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766326    5061 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766329    5061 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766332    5061 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766335    5061 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766338    5061 flags.go:64] FLAG: --port="10250"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766341    5061 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766344    5061 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ddbafb71124ee23e"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766347    5061 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766350    5061 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766354    5061 flags.go:64] FLAG: --register-node="true"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766357    5061 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766361    5061 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766365    5061 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766369    5061 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:40:54.774737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766372    5061 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766376    5061 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766379    5061 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766382    5061 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766385    5061 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766388    5061 flags.go:64] FLAG: --runonce="false"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766391    5061 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766395    5061 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766398    5061 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766401    5061 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766404    5061 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766407    5061 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766410    5061 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766413    5061 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766416    5061 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766419    5061 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766422    5061 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766425    5061 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766428    5061 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766431    5061 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766434    5061 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766439    5061 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766442    5061 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766445    5061 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766449    5061 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:40:54.775511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766452    5061 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766455    5061 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766458    5061 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766461    5061 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766464    5061 flags.go:64] FLAG: --v="2"
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766470    5061 flags.go:64] FLAG: --version="false"
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766476    5061 flags.go:64] FLAG: --vmodule=""
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766479    5061 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766483    5061 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766604    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766608    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766611    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766614    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766617    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766620    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766627    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766630    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766633    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766636    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766638    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766641    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:54.776189 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766643    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766646    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766648    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766651    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766654    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766657    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766659    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766662    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766665    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766667    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766670    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766672    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766675    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766678    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766680    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766683    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766686    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766690    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766693    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:54.776820 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766696    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766698    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766701    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766703    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766706    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766708    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766711    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766714    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766716    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766719    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766722    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766724    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766727    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766729    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766732    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766734    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766737    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766740    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766742    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766745    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:54.777454 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766747    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766750    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766753    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766755    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766758    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766760    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766763    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766765    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766768    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766770    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766774    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766777    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766780    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766783    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766785    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766788    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766790    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766793    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766796    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766798    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:54.778043 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766800    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766803    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766806    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766808    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766811    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766814    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766816    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766819    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766821    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766824    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766826    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766829    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766832    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766835    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766837    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766840    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766842    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766860    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766863    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:54.778721 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766866    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766868    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766871    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766875    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766879    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766883    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766888    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766891    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766894    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766897    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766899    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766902    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766905    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766907    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766910    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:54.779259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766913    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.766941    5061 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766984    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766989    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766992    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766996    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.766999    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767001    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767004    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767007    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767010    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767013    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767015    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767018    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767020    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767023    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:54.779723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767025    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767028    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767030    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767033    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767037    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767040    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767043    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767045    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767050    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767052    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767055    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767057    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767060    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767063    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767065    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767068    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767071    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767074    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767077    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:54.780373 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767079    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767082    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767084    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767087    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767090    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767092    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767095    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767098    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767101    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767103    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767106    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767109    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767111    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767114    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767117    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767119    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767122    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767127    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767129    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767132    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:54.780932 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767135    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767139    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767141    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767144    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767147    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767150    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767152    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767155    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767158    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767161    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767163    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767166    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767169    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767171    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767174    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767177    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767179    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767182    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767185    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767188    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:54.781445 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767191    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767193    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767196    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767198    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767201    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767203    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767206    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767209    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767211    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767215    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767218    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767221    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767223    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767227    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767230    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767232    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767235    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767238    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767240    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767243    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:54.781949 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767245    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767248    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767251    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767254    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767257    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767259    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767262    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767264    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767267    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767270    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767274    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767278    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.767282    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.772428    5061 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.772443    5061 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.772470    5061 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.772475    5061 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772533    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772539    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:54.782431 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772543    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772546    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772551    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772554    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772557    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772560    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772563    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772566    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772568    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772571    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772574    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772577    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772580    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772582    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772585    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772588    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772591    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772593    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772596    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772598    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:54.782918 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772602    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772606    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772610    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772613    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772616    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772619    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772626    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772629    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772631    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772634    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772637    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772639    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772642    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772644    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772647    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772650    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772653    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772655    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772658    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772660    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:54.783401 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772663    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772666    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772668    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772671    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772679    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772682    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772685    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772687    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772690    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772692    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772695    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772697    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772700    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772702    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772705    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772708    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772710    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772713    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772716    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772718    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:54.783980 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772721    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772733    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772736    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772739    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772741    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772744    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772747    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772750    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772752    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772755    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772757    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772761    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772763    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772766    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772769    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772771    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772774    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772783    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772786    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:54.784458 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772788    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772791    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772793    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772796    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772799    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772801    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772804    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772806    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772809    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772812    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772814    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772817    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772819    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772822    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772825    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772827    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772830    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772832    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772835    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:54.784933 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772838    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772840    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772854    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772857    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772860    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.772864    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.772893    5061 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773029    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773034    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773037    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773039    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773043    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773046    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773048    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773051    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:54.785386 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773054    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773057    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773059    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773062    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773064    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773067    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773069    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773072    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773074    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773077    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773080    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773082    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773085    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773087    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773090    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773093    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773096    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773098    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773101    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:54.785821 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773103    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773106    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773108    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773111    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773114    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773116    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773119    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773121    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773124    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773126    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773129    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773132    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773136    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773140    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773144    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773147    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773149    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773153    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773157    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:54.786331 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773160    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773163    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773167    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773170    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773172    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773175    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773180    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773182    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773185    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773188    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773191    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773193    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773196    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773198    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773201    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773204    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773206    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773210    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773215    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773219    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:54.786789 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773223    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773226    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773230    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773234    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773238    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773243    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773248    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773251    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773254    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773256    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773259    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773261    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773264    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773266    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773269    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773272    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773274    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773277    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773282    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773285    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:54.787286 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773288    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773290    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773293    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773296    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773299    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773301    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773304    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773307    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773309    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773312    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773315    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773318    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773321    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773323    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773326    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773328    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773331    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773334    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773337    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:54.787805 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773340    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.773365    5061 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773397    5061 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773402    5061 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773406    5061 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773410    5061 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773413    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773415    5061 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773418    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773421    5061 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773425    5061 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773429    5061 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773433    5061 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773436    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773439    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:54.788352 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773441    5061 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773444    5061 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773447    5061 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773449    5061 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773452    5061 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773455    5061 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773457    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773460    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773462    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773465    5061 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773468    5061 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773470    5061 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773473    5061 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773475    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773478    5061 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773480    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773483    5061 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773486    5061 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773489    5061 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773491    5061 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:54.788717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773494    5061 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773496    5061 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773498    5061 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773501    5061 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773504    5061 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773506    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773509    5061 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773512    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773516    5061 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773520    5061 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773523    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773525    5061 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773528    5061 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773531    5061 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773534    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773536    5061 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773539    5061 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773541    5061 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773544    5061 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:54.789281 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773547    5061 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773549    5061 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773552    5061 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773554    5061 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773557    5061 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773559    5061 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773562    5061 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773565    5061 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773567    5061 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773570    5061 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773573    5061 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773576    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773578    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773581    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773583    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773586    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773589    5061 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773592    5061 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773594    5061 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773597    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:54.789751 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773599    5061 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773603    5061 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773609    5061 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773612    5061 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773614    5061 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773617    5061 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773620    5061 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773622    5061 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773625    5061 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773627    5061 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773630    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773633    5061 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773635    5061 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773638    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773640    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773643    5061 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773646    5061 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773648    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773651    5061 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773653    5061 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:54.790577 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773656    5061 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773659    5061 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773661    5061 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773664    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773668    5061 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773671    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773674    5061 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773677    5061 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773679    5061 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773682    5061 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773684    5061 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773687    5061 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773689    5061 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:54.773693    5061 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.773742    5061 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.774720    5061 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.774766    5061 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.774948    5061 server.go:1014] "Starting client certificate rotation"
Oct 30 02:40:54.791307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.775108    5061 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:40:54.791764 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.775158    5061 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:35:13 +0000 UTC" deadline="2028-08-11 22:39:06.604379088 +0000 UTC"
Oct 30 02:40:54.791764 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.775185    5061 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15643h58m11.829195977s"
Oct 30 02:40:54.791764 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.775386    5061 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:54.791764 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.776502    5061 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:54.791764 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.778100    5061 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:40:54.791764 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.779823    5061 log.go:25] "Validated CRI v1 image API"
Oct 30 02:40:54.791764 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.780676    5061 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:40:54.791764 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.782798    5061 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 abd2dc97-6d78-40fa-b413-ff827ec7bbcb:/dev/nvme0n1p3 b03fa22c-7d29-46f6-b93e-aa3ed126fed2:/dev/nvme0n1p4]
Oct 30 02:40:54.792079 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.782817    5061 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03cfc82835843a9b804eab876a0e38b76a839fa5e125ba261219a303d233214f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03cfc82835843a9b804eab876a0e38b76a839fa5e125ba261219a303d233214f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1209f1414c30f5e2ef9460b8f1a063f56a5d0e68f861c9e020a5027a359d52a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1209f1414c30f5e2ef9460b8f1a063f56a5d0e68f861c9e020a5027a359d52a5/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b32445743e6f9c1ada4e2c374ae9fd6e4b3d9bce672b005023229590ec0e6c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b32445743e6f9c1ada4e2c374ae9fd6e4b3d9bce672b005023229590ec0e6c4/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21df96b6ed9f5503f4eff27cc64c39595412bd7860c5cb28f467edc77ffafa9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21df96b6ed9f5503f4eff27cc64c39595412bd7860c5cb28f467edc77ffafa9d/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/334191dd0df23d9916c528378d51f78f533f6ca3987f33bec6600101f20a4bd5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/334191dd0df23d9916c528378d51f78f533f6ca3987f33bec6600101f20a4bd5/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49c1fe3992694917dd43544066e8dcc194582356b399624e072c76ef961c2ae8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49c1fe3992694917dd43544066e8dcc194582356b399624e072c76ef961c2ae8/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c7cfaa83d792ca7515ddb62f7e21d968fa1fe46f5710a2f1d85f5c4f7c18190/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c7cfaa83d792ca7515ddb62f7e21d968fa1fe46f5710a2f1d85f5c4f7c18190/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ff5daf0bd925a2b6cd7e5be0e05585e1918f345b1ee300049371d9b2ce983ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ff5daf0bd925a2b6cd7e5be0e05585e1918f345b1ee300049371d9b2ce983ed/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6512af8f426058ddde7c7d59e9840651fffe4f32eb84ed2e5b6980b4ebaa07cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6512af8f426058ddde7c7d59e9840651fffe4f32eb84ed2e5b6980b4ebaa07cb/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71c7d22f628a11906ecbc82dceded6e9ec5c3b8f9b35f164eb4c0587129a7b87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71c7d22f628a11906ecbc82dceded6e9ec5c3b8f9b35f164eb4c0587129a7b87/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72213ea7d0fd40202430cd7d816217cc4b53a7a5d9dd3ddffc64d809d2d94e35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72213ea7d0fd40202430cd7d816217cc4b53a7a5d9dd3ddffc64d809d2d94e35/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72dd658f9795830c9a88e19578bdde58deb6ee4df059e5b8b0d8a1b205b11205/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72dd658f9795830c9a88e19578bdde58deb6ee4df059e5b8b0d8a1b205b11205/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82f612c6396937120581a6652be066f2edb8438f8e83271e6adc4bdee9ecea60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82f612c6396937120581a6652be066f2edb8438f8e83271e6adc4bdee9ecea60/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/832e66d95d736de2ef3040fe215a29f0111e963aafff9582eb57a504793ccf4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/832e66d95d736de2ef3040fe215a29f0111e963aafff9582eb57a504793ccf4a/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8deb66594f98df1f78e7d9d7080e4829b8fc21c2fab810da82082ba96b13712d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8deb66594f98df1f78e7d9d7080e4829b8fc21c2fab810da82082ba96b13712d/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b44207e9edbdf6ff9dab11296e0e8b14384e4a82771a47746ce3624c2991ee4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b44207e9edbdf6ff9dab11296e0e8b14384e4a82771a47746ce3624c2991ee4a/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9c9384fa29d31d5a3ff68f85599ee9ed4e2155eaefaf775e6a6ee97377bc2a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9c9384fa29d31d5a3ff68f85599ee9ed4e2155eaefaf775e6a6ee97377bc2a7/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c161763cc1a81d0ec45280c251660d734c6ee1894a175d8969c2ab828452effa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c161763cc1a81d0ec45280c251660d734c6ee1894a175d8969c2ab828452effa/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4d0066ee819fc8ce9c467b32fec17f97b20fc6fa6bc680a799b4613c5a342a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4d0066ee819fc8ce9c467b32fec17f97b20fc6fa6bc680a799b4613c5a342a1/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c588f69323516c04244fb14ee0a1da3b77a4ed46dba03f0f9a234ffef51e0a70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c588f69323516c04244fb14ee0a1da3b77a4ed46dba03f0f9a234ffef51e0a70/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7f726f259de8926e44f701d594b910c49baac0548c357964ad12baeb43074bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7f726f259de8926e44f701d594b910c49baac0548c357964ad12baeb43074bc/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e012fcef3d9f5dc2ef3cb13ffffb262b36d02196281a59fde68bfb10f6f4d84c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e012fcef3d9f5dc2ef3cb13ffffb262b36d02196281a59fde68bfb10f6f4d84c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e28b70462fc7a8a20c4d9cd368e1b33c4f9182fe166974975b81eb50d2201bde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e28b70462fc7a8a20c4d9cd368e1b33c4f9182fe166974975b81eb50d2201bde/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee041b160d660ef171b3245c331d118a56c65e2460759aeb7cbe3a92a6080b79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee041b160d660ef171b3245c331d118a56c65e2460759aeb7cbe3a92a6080b79/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5044505814979be024a50e25da60cb9036e68c8b6bb7cec66a5efa5053bf6eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5044505814979be024a50e25da60cb9036e68c8b6bb7cec66a5efa5053bf6eb/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe83b4c8f25d473ad47fc26249f164956aeb8979ce91f591f3d63d9778913bbe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe83b4c8f25d473ad47fc26249f164956aeb8979ce91f591f3d63d9778913bbe/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00b0139b-56b5-4daa-8fef-b86d0ac5a1d8/volumes/kubernetes.io~projected/kube-api-access-4fzsx:{mountpoint:/var/lib/kubelet/pods/00b0139b-56b5-4daa-8fef-b86d0ac5a1d8/volumes/kubernetes.io~projected/kube-api-access-4fzsx major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00bc8689-1f9b-40a7-9f19-972395d1bc69/volumes/kubernetes.io~projected/kube-api-access-wzrsj:{mountpoint:/var/lib/kubelet/pods/00bc8689-1f9b-40a7-9f19-972395d1bc69/volumes/kubernetes.io~projected/kube-api-access-wzrsj major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04ea065c-0b57-43ec-99be-989bb703b2c5/volumes/kubernetes.io~projected/kube-api-access-dw6qd:{mountpoint:/var/lib/kubelet/pods/04ea065c-0b57-43ec-99be-989bb703b2c5/volumes/kubernetes.io~projected/kube-api-access-dw6qd major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04ea065c-0b57-43ec-99be-989bb703b2c5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/04ea065c-0b57-43ec-99be-989bb703b2c5/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16395f11-a919-4dd3-9d21-3388d472e616/volumes/kubernetes.io~projected/kube-api-access-dt4f4:{mountpoint:/var/lib/kubelet/pods/16395f11-a919-4dd3-9d21-3388d472e616/volumes/kubernetes.io~projected/kube-api-access-dt4f4 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ea0be63-7705-40e8-b2a1-e48c39259c8b/volumes/kubernetes.io~projected/kube-api-access-pvgnv:{mountpoint:/var/lib/kubelet/pods/1ea0be63-7705-40e8-b2a1-e48c39259c8b/volumes/kubernetes.io~projected/kube-api-access-pvgnv major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ec92445-aa69-4865-9d4e-8a1423bf7073/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1ec92445-aa69-4865-9d4e-8a1423bf7073/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ec92445-aa69-4865-9d4e-8a1423bf7073/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1ec92445-aa69-4865-9d4e-8a1423bf7073/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ec92445-aa69-4865-9d4e-8a1423bf7073/volumes/kubernetes.io~projected/kube-api-access-wnvvc:{mountpoint:/var/lib/kubelet/pods/1ec92445-aa69-4865-9d4e-8a1423bf7073/volumes/kubernetes.io~projected/kube-api-access-wnvvc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd/volumes/kubernetes.io~projected/kube-api-access-ptjbq:{mountpoint:/var/lib/kubelet/pods/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd/volumes/kubernetes.io~projected/kube-api-access-ptjbq major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2967919e-d201-48ae-b388-85fe023e1d21/volumes/kubernetes.io~projected/kube-api-access-rmbq6:{mountpoint:/var/lib/kubelet/pods/2967919e-d201-48ae-b388-85fe023e1d21/volumes/kubernetes.io~projected/kube-api-access-rmbq6 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c04bf2f-3e60-442a-8639-10feae0a3f96/volumes/kubernetes.io~projected/kube-api-access-6qqnb:{mountpoint:/var/lib/kubelet/pods/2c04bf2f-3e60-442a-8639-10feae0a3f96/volumes/kubernetes.io~projected/kube-api-access-6qqnb major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33f38b3e-2ca5-40e1-ab36-f175b4a905fd/volumes/kubernetes.io~projected/kube-api-access-55nbl:{mountpoint:/var/lib/kubelet/pods/33f38b3e-2ca5-40e1-ab36-f175b4a905fd/volumes/kubernetes.io~projected/kube-api-access-55nbl major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8/volumes/kubernetes.io~projected/kube-api-access-9x9rq:{mountpoint:/var/lib/kubelet/pods/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8/volumes/kubernetes.io~projected/kube-api-access-9x9rq major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4844aa91-37e4-4cae-a380-af0000c768fc/volumes/kubernetes.io~projected/kube-api-access-526jr:{mountpoint:/var/lib/kubelet/pods/4844aa91-37e4-4cae-a380-af0000c768fc/volumes/kubernetes.io~projected/kube-api-access-526jr major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~projected/kube-api-access-dfg6j:{mountpoint:/var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~projected/kube-api-access-dfg6j major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51189dec-b0c6-430e-9768-8f9532bdb22a/volumes/kubernetes.io~projected/kube-api-access-cs586:{mountpoint:/var/lib/kubelet/pods/51189dec-b0c6-430e-9768-8f9532bdb22a/volumes/kubernetes.io~projected/kube-api-access-cs586 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d7330ef-3e60-407f-89ca-0c0c9e38c87d/volumes/kubernetes.io~projected/kube-api-access-zvz9z:{mountpoint:/var/lib/kubelet/pods/5d7330ef-3e60-407f-89ca-0c0c9e38c87d/volumes/kubernetes.io~projected/kube-api-access-zvz9z major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c28bc38-2c97-412f-89b0-f7a552b78373/volumes/kubernetes.io~projected/kube-api-access-lzh75:{mountpoint:/var/lib/kubelet/pods/6c28bc38-2c97-412f-89b0-f7a552b78373/volumes/kubernetes.io~projected/kube-api-access-lzh75 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c28bc38-2c97-412f-89b0-f7a552b78373/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/6c28bc38-2c97-412f-89b0-f7a552b78373/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c28bc38-2c97-412f-89b0-f7a552b78373/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/6c28bc38-2c97-412f-89b0-f7a552b78373/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7079cfbc-daff-4782-b3ca-e5718d1d1421/volumes/kubernetes.io~projected/kube-api-access-6qrc7:{mountpoint:/var/lib/kubelet/pods/7079cfbc-daff-4782-b3ca-e5718d1d1421/volumes/kubernetes.io~projected/kube-api-access-6qrc7 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7079cfbc-daff-4782-b3ca-e5718d1d1421/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/7079cfbc-daff-4782-b3ca-e5718d1d1421/volumes/kubernetes.io~secret/signing-key major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76243024-183c-42e6-89fd-ccd82d3571c4/volumes/kubernetes.io~projected/kube-api-access-5kk4c:{mountpoint:/var/lib/kubelet/pods/76243024-183c-42e6-89fd-ccd82d3571c4/volumes/kubernetes.io~projected/kube-api-access-5kk4c major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76243024-183c-42e6-89fd-ccd82d3571c4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/76243024-183c-42e6-89fd-ccd82d3571c4/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b6386c9-d170-4b89-bb41-9a0641207188/volumes/kubernetes.io~projected/kube-api-access-d7cpv:{mountpoint:/var/lib/kubelet/pods/7b6386c9-d170-4b89-bb41-9a0641207188/volumes/kubernetes.io~projected/kube-api-access-d7cpv major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b6386c9-d170-4b89-bb41-9a0641207188/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7b6386c9-d170-4b89-bb41-9a0641207188/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f9c5ebb-f865-483c-9ee3-a44c74aa3abe/volumes/kubernetes.io~projected/kube-api-access-kw4rt:{mountpoint:/var/lib/kubelet/pods/7f9c5ebb-f865-483c-9ee3-a44c74aa3abe/volumes/kubernetes.io~projected/kube-api-access-kw4rt major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/816aaac5-8df6-402e-9367-a2ea2f8ca686/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/816aaac5-8df6-402e-9367-a2ea2f8ca686/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/816aaac5-8df6-402e-9367-a2ea2f8ca686/volumes/kubernetes.io~projected/kube-api-access-6sbjc:{mountpoint:/var/lib/kubelet/pods/816aaac5-8df6-402e-9367-a2ea2f8ca686/volumes/kubernetes.io~projected/kube-api-access-6sbjc major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/816aaac5-8df6-402e-9367-a2ea2f8ca686/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/816aaac5-8df6-402e-9367-a2ea2f8ca686/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92515440-cc1f-4e4d-b45e-6afaff621108/volumes/kubernetes.io~projected/kube-api-access-qnvbf:{mountpoint:/var/lib/kubelet/pods/92515440-cc1f-4e4d-b45e-6afaff621108/volumes/kubernetes.io~projected/kube-api-access-qnvbf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/960da804-13fa-4a2b-8837-1dd6f8ca83c3/volumes/kubernetes.io~projected/kube-api-access-t6996:{mountpoint:/var/lib/kubelet/pods/960da804-13fa-4a2b-8837-1dd6f8ca83c3/volumes/kubernetes.io~projected/kube-api-access-t6996 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2bf413f-45b3-450c-b797-4e674e8da8ec/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a2bf413f-45b3-450c-b797-4e674e8da8ec/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af778722-0e75-4ddd-9819-d783cfa858e1/volumes/kubernetes.io~projected/kube-api-access-jjjrl:{mountpoint:/var/lib/kubelet/pods/af778722-0e75-4ddd-9819-d783cfa858e1/volumes/kubernetes.io~projected/kube-api-access-jjjrl major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8a2708-a7fe-41b0-8f63-7850df383c8f/volumes/kubernetes.io~projected/kube-api-access-ghvzs:{mountpoint:/var/lib/kubelet/pods/bf8a2708-a7fe-41b0-8f63-7850df383c8f/volumes/kubernetes.io~projected/kube-api-access-ghvzs major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1/volumes/kubernetes.io~projected/kube-api-access-6p5wx:{mountpoint:/var/lib/kubelet/pods/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1/volumes/kubernetes.io~projected/kube-api-access-6p5wx major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caeb5235-93eb-4b90-8372-1c59c807485f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/caeb5235-93eb-4b90-8372-1c59c807485f/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb4e2245-df10-43f4-a52a-433997bc0ca4/volumes/kubernetes.io~projected/kube-api-access-27s6x:{mountpoint:/var/lib/kubelet/pods/cb4e2245-df10-43f4-a52a-433997bc0ca4/volumes/kubernetes.io~projected/kube-api-access-27s6x major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~projected/kube-api-access-82nmv:{mountpoint:/var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~projected/kube-api-access-82nmv major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d786d081-b91d-475a-937b-021d17c210b1/volumes/kubernetes.io~projected/kube-api-access-b5wg8:{mountpoint:/var/lib/kubelet/pods/d786d081-b91d-475a-937b-021d17c210b1/volumes/kubernetes.io~projected/kube-api-access-b5wg8 major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0334aa2-bc83-440a-8781-60817828e1ab/volumes/kubernetes.io~projected/kube-api-access-tllq7:{mountpoint:/var/lib/kubelet/pods/e0334aa2-bc83-440a-8781-60817828e1ab/volumes/kubernetes.io~projected/kube-api-access-tllq7 major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0334aa2-bc83-440a-8781-60817828e1ab/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e0334aa2-bc83-440a-8781-60817828e1ab/volumes/kubernetes.io~secret/cert major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa5e5905-3e48-434f-8153-fd40123a614a/volumes/kubernetes.io~projected/kube-api-access-fvfg7:{mountpoint:/var/lib/kubelet/pods/fa5e5905-3e48-434f-8153-fd40123a614a/volumes/kubernetes.io~projected/kube-api-access-fvfg7 major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/19504c39f457204e37b35c6456064e53d087399bbf93686ce98811935dfe7780/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2388d7ace9feee684d9d1055c7adc3b23890b7c593421e022e2e25c04e43f5d7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b5a7e7f230a163bbc1aefe111700e8d93eb420f362916e264954fde0b0b9540d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/3edefc95148d7cac88eee95039a00dc0432184f6c1b97dea2cdc3fc93c210693/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/164501455f586fb9b6174ef28d60fb8e53caac2fcffd99fbe9d7eaaee38a1e83/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ceb0f1d75617121a4be93f7e58153573cc84efb3b8ee54b3f3fe3f824242c1e4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/cd97a0c0d49dab4c120423eb3385c0b29e5ab2cbd6ca3088c6c0edf968dd1c4a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3f206b0e300cc682df85ab596a84350a4ea59389e2398512c8214eae2e863627/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4cf93ebc127afa40d135683b7f97740a7f0b50c5ebb46d5643578df79e55012c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/0d943997f900b4e015187c039b22fc84f2e10274e08df8025fdac0f9a3746211/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/6e9af85184ce3df33a7b0a03223f4e70f9594deb8542d7684b8db39412cba19a/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/546c9f459bfaa7500f041e587281e0c464dee7415c51bbeaa3fdc3ac749e56b3/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a3ecf74ecd5c921ffda4f4b007aacd24b4854850a94f7ed246daaac825da8f36/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/611ab68d7cfc58ff84b5558ae8d1f8fe25c5a58b19dda52e127db568e214ffae/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e903b0b3151ec82a8f7196f5280bf88d7535fdd0ef66df6f26d3b5f82d8edb09/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cb6e34b43146d45d78d28e00a33841a69664d127fc2653ce47d4da07f7715f51/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b6094e9c8ac2ef18d7458206ecf1a43ba4137c96e697c42b5f04d809f86c4138/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/422377bf39aeb57af4b3eb5e5cfc46936e72c47da259df06d3aed4605a683726/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5df2ed246dcdcfaf61dc1d2942acbdd5ceb37d754b9b3bb4a765e9044e46eef3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/65be7ca25883483c4fa4ab0e2ab792c7108371e4559e283c3a4598d46192c306/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8576038d8e62f8b03ae7dddfdf6f0d4f215f5f12b00872b0164d45642dc767f7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/2cdc6bc2815a7efbd1cdd41547d754952e6557ae0305a5d23d580a846acb56a3/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b44a7f8b07605c57db0dd061b93eee90f0df10ed0915df896670cce4cfd4b759/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/7a0766fd4baeb797e81868bb39d9bc9ce22c6b41d989eeb5e76dc074bf07a87b/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/87be4e98c07595b25a1330957330f5d3fdd808b8217d06b5138cd7170a883835/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/07772b8b8248ee9ba50ba291265154b4f9c9d6fbf880051cc898ed26a611534b/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/8b1c9cdd2cc8f75866de127703cf469e1876e1fe7761d9e404883da4ca0acef6/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/eaff45e3be8c1553674b04ad47f12e232b8d10762c27eb5abc377b4c296cad8c/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/8b7098a5afea79602c8f64766c1fcd0de6432478f1f2eda78d92824dab1d3f24/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/8ea1d90bb93051859965276cd1f33718b3b3d7f89d385a4263d28424c37b8141/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/6c4375105a994179d9c171f641901f1a457aa8a17b3569f60e48485a4fae88a9/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/808f084cb5aa6b440203a9555447d393fe7ba9a5e3c31b3c03a8399975e15758/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/54ea848b8ac26a93c776e67c3cb619ba8e6e313a60ea85dea825e6e4642fcfe5/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/3d52b6915d71d2c53647c2515f334c5ac4b3ec6c527e93af876cafbfaa699b69/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/79bd52e3d08c23b5dcf13c2d827d21c2a12503788b24b24c264ca60bd2e74db0/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/e02f1cd4610e8e100387dbfca63a6ebe4e259ea72a77f7b4e12c93dadda9f7d7/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/ab383819a4624ffbcd265e5d99d44acf923d390b75a499682aa1c23972eb2b2f/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/fc97f49978b235a64532f5514d5dd2bbd8a8e52d180c815bba413ad74f6388b0/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/b2b0c2715c89f3b7f336447f455b68c1391648feaeefb17bdf9439f3e1ef47da/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/9aeebf6657a308c8b8599ade43642907b729846c81406a74e97637c440a7183b/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/eb5a86b3845994cf631bbeb3fe92d18edbaa677415ba43de6d458d7a42a876f7/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/fa783c04b95d4bf1cd87b30f94ff09da2fef73a9aa398df5c75846121781dbe8/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/c85e159b48027da7627011e616a68896a33f46ded1d7fe326c014520847373f6/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/87120e1edb4e9196cee04323127b5b8c8d2e7cc076ef1bd40b1a9ae69cb04411/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/ccfc34ff456f6b0d359d848778fb2dc38b949986f0112e24038764535652df7b/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/9485b51ca1399ebc9c3bc24efd85b675de899609b540c21914bd8be67e5778ea/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/6c97afb70ad99e46c92dd66d18aba2566e79fc7a199d9a55ae72fce540873990/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/bcea850b8cc739215b11eb898b671c03238654cd3ba87b546724741fc4d15b3e/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/c8baf970d2ba21c56bccae756e0fd1fec6958125081498153c3807ae5004e827/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/0b641b49371d68dd558b739aea5644f55f756bab580ee7cd29818564efa52456/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/7902c22481f551874b627542a8514e9f9b2aa5e415db272dd5c702134c1fd973/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/92d133e6eead87e573cf929ad347247f9256f6552128b153a4aa49192d366c62/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b74b0619a2bca76c4a9b318c755bfb6557cd61e8052a8ca85f4bab25d31d6651/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/71f69b78a1c1abff020212022a1e13e401262a653fabb7321f961fe079c7e4d6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/220711e91851f52d39260dca8418ddc852242e0afd1a8b8319557424f7307ec0/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/845dcfe193fda1228baddcc339ee8b198369de1b28edb700044ec2448e64b2f8/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/40b1d4b297e2b7d15676c1d62eadfb88a489650d83f171badbd9b000c91da1ec/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/46c551f2f5c6176102ae4805b6c600345d9bd09a2acd159c3a114c6694afd02e/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a2560fa8aa5408fee515407de2e230a408467bc2680b40fb29b0ceab76b2c810/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/35762099c9dff9a8349bd4ba6a4d1c47531a5282a78849fb0d0b153e14aa7001/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/e5b28f820149dc97c399a28586508a2dd8682f2d8fa20801cf7fdaba21a491ea/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9a64470e84d849cd90e552f659e4c2b190fe92b2721d1a161fbd2023b911fa73/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 02:40:54.793221 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.792818    5061 manager.go:217] Machine: {Timestamp:2026-10-30 02:40:54.792359503 +0000 UTC m=+0.094537859 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108769 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2785ac5795e731f83b8784b9ee237b SystemUUID:ec2785ac-5795-e731-f83b-8784b9ee237b BootID:4fabfbc6-19a9-403c-96b3-1a00cf354f57 Filesystems:[{Device:/var/lib/kubelet/pods/816aaac5-8df6-402e-9367-a2ea2f8ca686/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8a2708-a7fe-41b0-8f63-7850df383c8f/volumes/kubernetes.io~projected/kube-api-access-ghvzs DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7079cfbc-daff-4782-b3ca-e5718d1d1421/volumes/kubernetes.io~projected/kube-api-access-6qrc7 DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8/volumes/kubernetes.io~projected/kube-api-access-9x9rq DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9c9384fa29d31d5a3ff68f85599ee9ed4e2155eaefaf775e6a6ee97377bc2a7/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7079cfbc-daff-4782-b3ca-e5718d1d1421/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f9c5ebb-f865-483c-9ee3-a44c74aa3abe/volumes/kubernetes.io~projected/kube-api-access-kw4rt DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1ec92445-aa69-4865-9d4e-8a1423bf7073/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2c04bf2f-3e60-442a-8639-10feae0a3f96/volumes/kubernetes.io~projected/kube-api-access-6qqnb DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7b6386c9-d170-4b89-bb41-9a0641207188/volumes/kubernetes.io~projected/kube-api-access-d7cpv DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa5e5905-3e48-434f-8153-fd40123a614a/volumes/kubernetes.io~projected/kube-api-access-fvfg7 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a2bf413f-45b3-450c-b797-4e674e8da8ec/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb4e2245-df10-43f4-a52a-433997bc0ca4/volumes/kubernetes.io~projected/kube-api-access-27s6x DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b6386c9-d170-4b89-bb41-9a0641207188/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b32445743e6f9c1ada4e2c374ae9fd6e4b3d9bce672b005023229590ec0e6c4/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/334191dd0df23d9916c528378d51f78f533f6ca3987f33bec6600101f20a4bd5/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/816aaac5-8df6-402e-9367-a2ea2f8ca686/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7f726f259de8926e44f701d594b910c49baac0548c357964ad12baeb43074bc/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ea0be63-7705-40e8-b2a1-e48c39259c8b/volumes/kubernetes.io~projected/kube-api-access-pvgnv DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4d0066ee819fc8ce9c467b32fec17f97b20fc6fa6bc680a799b4613c5a342a1/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e012fcef3d9f5dc2ef3cb13ffffb262b36d02196281a59fde68bfb10f6f4d84c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1ec92445-aa69-4865-9d4e-8a1423bf7073/volumes/kubernetes.io~projected/kube-api-access-wnvvc DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0334aa2-bc83-440a-8781-60817828e1ab/volumes/kubernetes.io~projected/kube-api-access-tllq7 DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/00b0139b-56b5-4daa-8fef-b86d0ac5a1d8/volumes/kubernetes.io~projected/kube-api-access-4fzsx DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1209f1414c30f5e2ef9460b8f1a063f56a5d0e68f861c9e020a5027a359d52a5/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d7330ef-3e60-407f-89ca-0c0c9e38c87d/volumes/kubernetes.io~projected/kube-api-access-zvz9z DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6c28bc38-2c97-412f-89b0-f7a552b78373/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d786d081-b91d-475a-937b-021d17c210b1/volumes/kubernetes.io~projected/kube-api-access-b5wg8 DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76243024-183c-42e6-89fd-ccd82d3571c4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6c28bc38-2c97-412f-89b0-f7a552b78373/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2967919e-d201-48ae-b388-85fe023e1d21/volumes/kubernetes.io~projected/kube-api-access-rmbq6 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51189dec-b0c6-430e-9768-8f9532bdb22a/volumes/kubernetes.io~projected/kube-api-access-cs586 DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ff5daf0bd925a2b6cd7e5be0e05585e1918f345b1ee300049371d9b2ce983ed/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33f38b3e-2ca5-40e1-ab36-f175b4a905fd/volumes/kubernetes.io~projected/kube-api-access-55nbl DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c161763cc1a81d0ec45280c251660d734c6ee1894a175d8969c2ab828452effa/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b44207e9edbdf6ff9dab11296e0e8b14384e4a82771a47746ce3624c2991ee4a/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5044505814979be024a50e25da60cb9036e68c8b6bb7cec66a5efa5053bf6eb/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c588f69323516c04244fb14ee0a1da3b77a4ed46dba03f0f9a234ffef51e0a70/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82f612c6396937120581a6652be066f2edb8438f8e83271e6adc4bdee9ecea60/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04ea065c-0b57-43ec-99be-989bb703b2c5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/16395f11-a919-4dd3-9d21-3388d472e616/volumes/kubernetes.io~projected/kube-api-access-dt4f4 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1/volumes/kubernetes.io~projected/kube-api-access-6p5wx DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8deb66594f98df1f78e7d9d7080e4829b8fc21c2fab810da82082ba96b13712d/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6512af8f426058ddde7c7d59e9840651fffe4f32eb84ed2e5b6980b4ebaa07cb/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/832e66d95d736de2ef3040fe215a29f0111e963aafff9582eb57a504793ccf4a/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03cfc82835843a9b804eab876a0e38b76a839fa5e125ba261219a303d233214f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee041b160d660ef171b3245c331d118a56c65e2460759aeb7cbe3a92a6080b79/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04ea065c-0b57-43ec-99be-989bb703b2c5/volumes/kubernetes.io~projected/kube-api-access-dw6qd DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c28bc38-2c97-412f-89b0-f7a552b78373/volumes/kubernetes.io~projected/kube-api-access-lzh75 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4844aa91-37e4-4cae-a380-af0000c768fc/volumes/kubernetes.io~projected/kube-api-access-526jr DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~projected/kube-api-access-82nmv DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0334aa2-bc83-440a-8781-60817828e1ab/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71c7d22f628a11906ecbc82dceded6e9ec5c3b8f9b35f164eb4c0587129a7b87/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1ec92445-aa69-4865-9d4e-8a1423bf7073/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72dd658f9795830c9a88e19578bdde58deb6ee4df059e5b8b0d8a1b205b11205/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76243024-183c-42e6-89fd-ccd82d3571c4/volumes/kubernetes.io~projected/kube-api-access-5kk4c DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd/volumes/kubernetes.io~projected/kube-api-access-ptjbq DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c7cfaa83d792ca7515ddb62f7e21d968fa1fe46f5710a2f1d85f5c4f7c18190/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72213ea7d0fd40202430cd7d816217cc4b53a7a5d9dd3ddffc64d809d2d94e35/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e28b70462fc7a8a20c4d9cd368e1b33c4f9182fe166974975b81eb50d2201bde/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/caeb5235-93eb-4b90-8372-1c59c807485f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92515440-cc1f-4e4d-b45e-6afaff621108/volumes/kubernetes.io~projected/kube-api-access-qnvbf DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/00bc8689-1f9b-40a7-9f19-972395d1bc69/volumes/kubernetes.io~projected/kube-api-access-wzrsj DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21df96b6ed9f5503f4eff27cc64c39595412bd7860c5cb28f467edc77ffafa9d/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af778722-0e75-4ddd-9819-d783cfa858e1/volumes/kubernetes.io~projected/kube-api-access-jjjrl DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/816aaac5-8df6-402e-9367-a2ea2f8ca686/volumes/kubernetes.io~projected/kube-api-access-6sbjc DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49c1fe3992694917dd43544066e8dcc194582356b399624e072c76ef961c2ae8/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes/kubernetes.io~projected/kube-api-access-dfg6j DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/960da804-13fa-4a2b-8837-1dd6f8ca83c3/volumes/kubernetes.io~projected/kube-api-access-t6996 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe83b4c8f25d473ad47fc26249f164956aeb8979ce91f591f3d63d9778913bbe/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1209f1414c30f5e MacAddress:96:7e:fa:4e:ad:99 Speed:10000 Mtu:8901} {Name:1b32445743e6f9c MacAddress:ee:3c:b8:2e:25:11 Speed:10000 Mtu:8901} {Name:21df96b6ed9f550 MacAddress:a2:15:1c:9c:75:25 Speed:10000 Mtu:8901} {Name:4c7cfaa83d792ca MacAddress:7a:0d:e4:31:99:3c Speed:10000 Mtu:8901} {Name:5ff5daf0bd925a2 MacAddress:8e:41:dd:47:3b:17 Speed:10000 Mtu:8901} {Name:6512af8f426058d MacAddress:7e:39:6b:f1:94:4f Speed:10000 Mtu:8901} {Name:71c7d22f628a119 MacAddress:ae:a1:5f:37:4f:b9 Speed:10000 Mtu:8901} {Name:72dd658f9795830 MacAddress:ca:09:9d:0f:fc:b9 Speed:10000 Mtu:8901} {Name:82f612c63969371 MacAddress:8a:28:06:f2:ce:d3 Speed:10000 Mtu:8901} {Name:832e66d95d736de MacAddress:92:e8:b6:26:4d:a2 Speed:10000 Mtu:8901} {Name:8deb66594f98df1 MacAddress:fe:f3:8b:e9:18:5f Speed:10000 Mtu:8901} {Name:b44207e9edbdf6f MacAddress:ba:eb:f2:5a:dd:e7 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d9:e9:7e:f7 Speed:0 Mtu:9001} {Name:br-int MacAddress:26:32:98:bd:e9:76 Speed:0 Mtu:8901} {Name:c161763cc1a81d0 MacAddress:be:5f:2e:9f:ba:77 Speed:10000 Mtu:8901} {Name:c4d0066ee819fc8 MacAddress:e6:af:2f:f2:84:2b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d9:e9:7e:f7 Speed:0 Mtu:9001} {Name:f5044505814979b MacAddress:0a:2c:d4:c4:d9:fb Speed:10000 Mtu:8901} {Name:fe83b4c8f25d473 MacAddress:a6:34:04:a7:c3:20 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a2:aa:8e:9a:14:ae Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d6:5c:1c:7a:af:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:40:54.793470 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.793226    5061 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:40:54.793470 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.793273    5061 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:40:54.793576 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.793549    5061 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:40:54.793702 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.793577    5061 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-238.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:40:54.793749 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.793704    5061 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:40:54.793749 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.793712    5061 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:40:54.793749 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.793722    5061 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:54.793749 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.793740    5061 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:54.793866 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.793757    5061 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:40:54.793931 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.793920    5061 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:40:54.794028 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.794019    5061 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:40:54.794065 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.794052    5061 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:40:54.794065 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.794065    5061 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:40:54.794139 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.794073    5061 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:40:54.794139 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.794082    5061 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:40:54.794139 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.794089    5061 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:40:54.794842 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.794826    5061 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:40:54.794842 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.794838    5061 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:54.795272 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.795248    5061 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:40:54.797840 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.797824    5061 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:40:54.797840 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.797838    5061 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:54.797962 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.797951    5061 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 02:40:54.798155 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798144    5061 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:40:54.798185 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798174    5061 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:40:54.798260 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798252    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:40:54.798293 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798266    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:40:54.798293 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798272    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:40:54.798293 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798279    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:40:54.798293 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798292    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:40:54.798397 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798306    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:40:54.798397 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798319    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:40:54.798397 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798329    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:40:54.798397 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798342    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:40:54.798397 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798351    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:40:54.798397 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798364    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:40:54.798397 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798378    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:40:54.798575 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798405    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:40:54.798575 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.798414    5061 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:40:54.801836 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.801825    5061 server.go:1263] "Started kubelet"
Oct 30 02:40:54.801950 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.801922    5061 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:40:54.802338 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.802050    5061 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:40:54.802338 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.802142    5061 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:40:54.802464 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.802445    5061 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:40:54.802536 ip-10-0-136-238 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:40:54.806939 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.804974    5061 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:40:54.807411 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.807396    5061 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:40:54.808257 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.807887    5061 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:40:54.808257 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.807983    5061 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:15 +0000 UTC" deadline="2028-06-17 07:45:58.132195216 +0000 UTC"
Oct 30 02:40:54.808257 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.808040    5061 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14309h5m3.324159378s"
Oct 30 02:40:54.808257 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.808096    5061 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:40:54.808257 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.808106    5061 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:40:54.808257 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.808197    5061 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:40:54.808257 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:54.808196    5061 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-238.ec2.internal\" not found"
Oct 30 02:40:54.809979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.809958    5061 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:40:54.809979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.809979    5061 factory.go:55] Registering systemd factory
Oct 30 02:40:54.810118 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.809989    5061 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:40:54.810782 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.810758    5061 factory.go:153] Registering CRI-O factory
Oct 30 02:40:54.810782 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.810776    5061 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:40:54.810958 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.810794    5061 factory.go:103] Registering Raw factory
Oct 30 02:40:54.810958 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.810807    5061 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:40:54.811194 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811179    5061 manager.go:319] Starting recovery of all containers
Oct 30 02:40:54.811300 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811238    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16395f11-a919-4dd3-9d21-3388d472e616" volumeName="kubernetes.io/configmap/16395f11-a919-4dd3-9d21-3388d472e616-config-volume" seLinuxMountContext=""
Oct 30 02:40:54.811403 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811304    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16395f11-a919-4dd3-9d21-3388d472e616" volumeName="kubernetes.io/projected/16395f11-a919-4dd3-9d21-3388d472e616-kube-api-access-dt4f4" seLinuxMountContext=""
Oct 30 02:40:54.811403 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811317    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33f38b3e-2ca5-40e1-ab36-f175b4a905fd" volumeName="kubernetes.io/empty-dir/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-data-volume" seLinuxMountContext=""
Oct 30 02:40:54.811403 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811328    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="816aaac5-8df6-402e-9367-a2ea2f8ca686" volumeName="kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-config" seLinuxMountContext=""
Oct 30 02:40:54.811403 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811351    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00bc8689-1f9b-40a7-9f19-972395d1bc69" volumeName="kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj" seLinuxMountContext=""
Oct 30 02:40:54.811403 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811361    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ec92445-aa69-4865-9d4e-8a1423bf7073" volumeName="kubernetes.io/projected/1ec92445-aa69-4865-9d4e-8a1423bf7073-kube-api-access-wnvvc" seLinuxMountContext=""
Oct 30 02:40:54.811403 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811371    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c04bf2f-3e60-442a-8639-10feae0a3f96" volumeName="kubernetes.io/projected/2c04bf2f-3e60-442a-8639-10feae0a3f96-kube-api-access-6qqnb" seLinuxMountContext=""
Oct 30 02:40:54.811403 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811382    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" volumeName="kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 02:40:54.811403 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811396    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7079cfbc-daff-4782-b3ca-e5718d1d1421" volumeName="kubernetes.io/projected/7079cfbc-daff-4782-b3ca-e5718d1d1421-kube-api-access-6qrc7" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811408    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76243024-183c-42e6-89fd-ccd82d3571c4" volumeName="kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-service-ca-bundle" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811421    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76243024-183c-42e6-89fd-ccd82d3571c4" volumeName="kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-tmp" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811435    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="816aaac5-8df6-402e-9367-a2ea2f8ca686" volumeName="kubernetes.io/secret/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811447    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92515440-cc1f-4e4d-b45e-6afaff621108" volumeName="kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-multus-daemon-config" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811458    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8a2708-a7fe-41b0-8f63-7850df383c8f" volumeName="kubernetes.io/projected/bf8a2708-a7fe-41b0-8f63-7850df383c8f-kube-api-access-ghvzs" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811495    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16395f11-a919-4dd3-9d21-3388d472e616" volumeName="kubernetes.io/empty-dir/16395f11-a919-4dd3-9d21-3388d472e616-tmp-dir" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811506    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="215f867e-a2ba-4b0d-ada3-6182c5e1a8fd" volumeName="kubernetes.io/configmap/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-config" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811520    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" volumeName="kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-bound-sa-token" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811532    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f9c5ebb-f865-483c-9ee3-a44c74aa3abe" volumeName="kubernetes.io/projected/7f9c5ebb-f865-483c-9ee3-a44c74aa3abe-kube-api-access-kw4rt" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811543    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af778722-0e75-4ddd-9819-d783cfa858e1" volumeName="kubernetes.io/projected/af778722-0e75-4ddd-9819-d783cfa858e1-kube-api-access-jjjrl" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811553    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9aa32ac-0748-4d2f-820d-8a15ec78cdb1" volumeName="kubernetes.io/configmap/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-telemetry-config" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811566    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd57c8d3-5a2c-4068-b04a-901e5795b815" volumeName="kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-certificates" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811578    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2967919e-d201-48ae-b388-85fe023e1d21" volumeName="kubernetes.io/projected/2967919e-d201-48ae-b388-85fe023e1d21-kube-api-access-rmbq6" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811590    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47e2d60c-0e3c-4c87-9198-b2d5bc0122c8" volumeName="kubernetes.io/secret/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-serving-cert" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811603    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d7330ef-3e60-407f-89ca-0c0c9e38c87d" volumeName="kubernetes.io/projected/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-kube-api-access-zvz9z" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811642    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb4e2245-df10-43f4-a52a-433997bc0ca4" volumeName="kubernetes.io/projected/cb4e2245-df10-43f4-a52a-433997bc0ca4-kube-api-access-27s6x" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811655    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d786d081-b91d-475a-937b-021d17c210b1" volumeName="kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811667    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0334aa2-bc83-440a-8781-60817828e1ab" volumeName="kubernetes.io/projected/e0334aa2-bc83-440a-8781-60817828e1ab-kube-api-access-tllq7" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811681    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa5e5905-3e48-434f-8153-fd40123a614a" volumeName="kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:40:54.811705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811701    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" volumeName="kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-kube-api-access-dfg6j" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811714    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b6386c9-d170-4b89-bb41-9a0641207188" volumeName="kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-trusted-ca" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811740    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0334aa2-bc83-440a-8781-60817828e1ab" volumeName="kubernetes.io/secret/e0334aa2-bc83-440a-8781-60817828e1ab-cert" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811763    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ec92445-aa69-4865-9d4e-8a1423bf7073" volumeName="kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-tuned" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811775    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="215f867e-a2ba-4b0d-ada3-6182c5e1a8fd" volumeName="kubernetes.io/projected/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-kube-api-access-ptjbq" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811787    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b6386c9-d170-4b89-bb41-9a0641207188" volumeName="kubernetes.io/secret/7b6386c9-d170-4b89-bb41-9a0641207188-serving-cert" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811815    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="960da804-13fa-4a2b-8837-1dd6f8ca83c3" volumeName="kubernetes.io/configmap/960da804-13fa-4a2b-8837-1dd6f8ca83c3-iptables-alerter-script" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811828    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd57c8d3-5a2c-4068-b04a-901e5795b815" volumeName="kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-bound-sa-token" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811840    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa5e5905-3e48-434f-8153-fd40123a614a" volumeName="kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811897    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ea065c-0b57-43ec-99be-989bb703b2c5" volumeName="kubernetes.io/projected/04ea065c-0b57-43ec-99be-989bb703b2c5-kube-api-access-dw6qd" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811910    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c28bc38-2c97-412f-89b0-f7a552b78373" volumeName="kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-default-certificate" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811926    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92515440-cc1f-4e4d-b45e-6afaff621108" volumeName="kubernetes.io/projected/92515440-cc1f-4e4d-b45e-6afaff621108-kube-api-access-qnvbf" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811939    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00b0139b-56b5-4daa-8fef-b86d0ac5a1d8" volumeName="kubernetes.io/projected/00b0139b-56b5-4daa-8fef-b86d0ac5a1d8-kube-api-access-4fzsx" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811950    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ea0be63-7705-40e8-b2a1-e48c39259c8b" volumeName="kubernetes.io/projected/1ea0be63-7705-40e8-b2a1-e48c39259c8b-kube-api-access-pvgnv" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811962    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c28bc38-2c97-412f-89b0-f7a552b78373" volumeName="kubernetes.io/projected/6c28bc38-2c97-412f-89b0-f7a552b78373-kube-api-access-lzh75" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811974    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="816aaac5-8df6-402e-9367-a2ea2f8ca686" volumeName="kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.811984    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="960da804-13fa-4a2b-8837-1dd6f8ca83c3" volumeName="kubernetes.io/projected/960da804-13fa-4a2b-8837-1dd6f8ca83c3-kube-api-access-t6996" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812016    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caeb5235-93eb-4b90-8372-1c59c807485f" volumeName="kubernetes.io/secret/caeb5235-93eb-4b90-8372-1c59c807485f-agent-certs" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812036    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4844aa91-37e4-4cae-a380-af0000c768fc" volumeName="kubernetes.io/projected/4844aa91-37e4-4cae-a380-af0000c768fc-kube-api-access-526jr" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812047    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76243024-183c-42e6-89fd-ccd82d3571c4" volumeName="kubernetes.io/secret/76243024-183c-42e6-89fd-ccd82d3571c4-serving-cert" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812058    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92515440-cc1f-4e4d-b45e-6afaff621108" volumeName="kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812089    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af778722-0e75-4ddd-9819-d783cfa858e1" volumeName="kubernetes.io/configmap/af778722-0e75-4ddd-9819-d783cfa858e1-serviceca" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812102    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd57c8d3-5a2c-4068-b04a-901e5795b815" volumeName="kubernetes.io/empty-dir/cd57c8d3-5a2c-4068-b04a-901e5795b815-ca-trust-extracted" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812114    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d786d081-b91d-475a-937b-021d17c210b1" volumeName="kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-service-ca-bundle" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812127    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa5e5905-3e48-434f-8153-fd40123a614a" volumeName="kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812141    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ea065c-0b57-43ec-99be-989bb703b2c5" volumeName="kubernetes.io/secret/04ea065c-0b57-43ec-99be-989bb703b2c5-serving-cert" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812157    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ec92445-aa69-4865-9d4e-8a1423bf7073" volumeName="kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-tmp" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812179    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51189dec-b0c6-430e-9768-8f9532bdb22a" volumeName="kubernetes.io/projected/51189dec-b0c6-430e-9768-8f9532bdb22a-kube-api-access-cs586" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812208    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7079cfbc-daff-4782-b3ca-e5718d1d1421" volumeName="kubernetes.io/configmap/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-cabundle" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812219    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7079cfbc-daff-4782-b3ca-e5718d1d1421" volumeName="kubernetes.io/secret/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-key" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812232    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b6386c9-d170-4b89-bb41-9a0641207188" volumeName="kubernetes.io/projected/7b6386c9-d170-4b89-bb41-9a0641207188-kube-api-access-d7cpv" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812242    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caeb5235-93eb-4b90-8372-1c59c807485f" volumeName="kubernetes.io/configmap/caeb5235-93eb-4b90-8372-1c59c807485f-konnectivity-ca" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812252    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="816aaac5-8df6-402e-9367-a2ea2f8ca686" volumeName="kubernetes.io/projected/816aaac5-8df6-402e-9367-a2ea2f8ca686-kube-api-access-6sbjc" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812290    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ea065c-0b57-43ec-99be-989bb703b2c5" volumeName="kubernetes.io/configmap/04ea065c-0b57-43ec-99be-989bb703b2c5-config" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812335    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2bf413f-45b3-450c-b797-4e674e8da8ec" volumeName="kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812348    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9aa32ac-0748-4d2f-820d-8a15ec78cdb1" volumeName="kubernetes.io/projected/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-kube-api-access-6p5wx" seLinuxMountContext=""
Oct 30 02:40:54.812365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812378    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd57c8d3-5a2c-4068-b04a-901e5795b815" volumeName="kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-trusted-ca" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812393    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd57c8d3-5a2c-4068-b04a-901e5795b815" volumeName="kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-kube-api-access-82nmv" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812414    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d786d081-b91d-475a-937b-021d17c210b1" volumeName="kubernetes.io/empty-dir/d786d081-b91d-475a-937b-021d17c210b1-archives-path" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812437    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d786d081-b91d-475a-937b-021d17c210b1" volumeName="kubernetes.io/projected/d786d081-b91d-475a-937b-021d17c210b1-kube-api-access-b5wg8" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812451    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47e2d60c-0e3c-4c87-9198-b2d5bc0122c8" volumeName="kubernetes.io/configmap/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-config" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812465    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" volumeName="kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-certificates" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812490    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" volumeName="kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-trusted-ca" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812504    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" volumeName="kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-installation-pull-secrets" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812515    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="816aaac5-8df6-402e-9367-a2ea2f8ca686" volumeName="kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-env-overrides" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812565    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" volumeName="kubernetes.io/empty-dir/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-ca-trust-extracted" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812588    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c28bc38-2c97-412f-89b0-f7a552b78373" volumeName="kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-stats-auth" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812600    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76243024-183c-42e6-89fd-ccd82d3571c4" volumeName="kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812623    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76243024-183c-42e6-89fd-ccd82d3571c4" volumeName="kubernetes.io/projected/76243024-183c-42e6-89fd-ccd82d3571c4-kube-api-access-5kk4c" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812636    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd57c8d3-5a2c-4068-b04a-901e5795b815" volumeName="kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812680    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa5e5905-3e48-434f-8153-fd40123a614a" volumeName="kubernetes.io/projected/fa5e5905-3e48-434f-8153-fd40123a614a-kube-api-access-fvfg7" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812721    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33f38b3e-2ca5-40e1-ab36-f175b4a905fd" volumeName="kubernetes.io/projected/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-api-access-55nbl" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812746    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47e2d60c-0e3c-4c87-9198-b2d5bc0122c8" volumeName="kubernetes.io/projected/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-kube-api-access-9x9rq" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812769    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69caed51-464d-4dab-b4a4-b4ff77da5e4e" volumeName="kubernetes.io/configmap/69caed51-464d-4dab-b4a4-b4ff77da5e4e-nginx-conf" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812838    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76243024-183c-42e6-89fd-ccd82d3571c4" volumeName="kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-snapshots" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812866    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b6386c9-d170-4b89-bb41-9a0641207188" volumeName="kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-config" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812878    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd57c8d3-5a2c-4068-b04a-901e5795b815" volumeName="kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-installation-pull-secrets" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812901    5061 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d7330ef-3e60-407f-89ca-0c0c9e38c87d" volumeName="kubernetes.io/empty-dir/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-tmp-dir" seLinuxMountContext=""
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812915    5061 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:40:54.813689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.812923    5061 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:40:54.834352 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.834324    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:40:54.838279 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.837323    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:40:54.838279 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.837491    5061 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:40:54.838499 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.838480    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:40:54.839108 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.839081    5061 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:40:54.839108 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.839104    5061 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:40:54.839277 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.839124    5061 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:40:54.839277 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:54.839165    5061 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 02:40:54.841686 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.841664    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:40:54.845256 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.845237    5061 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 02:40:54.846497 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.846162    5061 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:54.846497 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.846322    5061 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:54.857170 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.857149    5061 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:54.857409 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.857388    5061 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:54.864445 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.864412    5061 generic.go:356] "Generic (PLEG): container finished" podID="6521a54728e46ede7b6de87a0d41821f" containerID="370411047ee34b250d3138bf063e0fa9b97bd5798ad5ae523689dd7f38bdf09d" exitCode=0
Oct 30 02:40:54.874334 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.874315    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:40:54.874415 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.874348    5061 generic.go:356] "Generic (PLEG): container finished" podID="7b6386c9-d170-4b89-bb41-9a0641207188" containerID="b964cb6f1289891b21a2f16154013500efe4d5530fd822329a568441f26a926f" exitCode=255
Oct 30 02:40:54.881700 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.881680    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:40:54.882046 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.882019    5061 generic.go:356] "Generic (PLEG): container finished" podID="816aaac5-8df6-402e-9367-a2ea2f8ca686" containerID="5651f2e0747b4afda8b7cafc7fa07f94b759a8fae4574fc59b5ae22bf7339f80" exitCode=1
Oct 30 02:40:54.896385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.896336    5061 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="a4b2d7f9c00b257b2eb6daf2576131d1b06096d8851081b339aeee6d1a048f97" exitCode=0
Oct 30 02:40:54.896385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.896359    5061 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="f4f4d19614e511fe50e77cbe55ea371649eba756aef1307273b08ea9b5964a8e" exitCode=0
Oct 30 02:40:54.896385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.896370    5061 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="b02e3b09210c9252795d52a7669f09bf2c273d7bd18ac1ece2aabb20826f64e8" exitCode=0
Oct 30 02:40:54.896385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.896378    5061 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="66617e43942161109e8464ff21a4b3cecfa6e2de6e9ebbd342bd19d2f6fa2a3d" exitCode=0
Oct 30 02:40:54.896385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.896383    5061 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="d484f28209b3bf91016b3e211aceeeb6d6bda9bde6dbcef8d51d1314395f7a50" exitCode=0
Oct 30 02:40:54.896385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.896388    5061 generic.go:356] "Generic (PLEG): container finished" podID="fa5e5905-3e48-434f-8153-fd40123a614a" containerID="60aae2bd1ce494349d5ca1f43ada934fb6927531777bdd062b77196a7686481d" exitCode=0
Oct 30 02:40:54.913316 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.913297    5061 manager.go:324] Recovery completed
Oct 30 02:40:54.934606 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.934585    5061 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:40:54.934606 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.934599    5061 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:40:54.934713 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.934615    5061 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:40:54.934745 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.934729    5061 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 02:40:54.935174 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.934739    5061 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 02:40:54.935220 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.935177    5061 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 02:40:54.935220 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.935183    5061 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 02:40:54.935220 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.935189    5061 policy_none.go:50] "Start"
Oct 30 02:40:54.935220 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.935198    5061 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:40:54.935220 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.935210    5061 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:40:54.935346 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.935337    5061 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 02:40:54.935346 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.935344    5061 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 02:40:54.935400 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.935350    5061 policy_none.go:44] "Start"
Oct 30 02:40:54.939565 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:54.939550    5061 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 02:40:54.939701 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.939683    5061 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:40:54.942503 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.942487    5061 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:40:54.942585 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:54.942525    5061 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:40:54.942585 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.942542    5061 server.go:87] "Starting device plugin registration server"
Oct 30 02:40:54.942799 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.942785    5061 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:40:54.942914 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.942807    5061 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:40:54.942914 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.942884    5061 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:40:54.943007 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.942978    5061 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:40:54.943055 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.943013    5061 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:40:54.943055 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.943018    5061 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:40:54.943141 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.943060    5061 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:40:54.943005056Z","UUID":"7e7d94a1-1a54-42c4-a34a-840b86c5a5e0","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:40:54.944393 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:54.944368    5061 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:40:54.944483 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.944466    5061 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:40:54.944483 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:54.944487    5061 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:40:55.044946 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.044921    5061 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:55.046166 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.046144    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:55.046264 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.046175    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:55.046264 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.046185    5061 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-238.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:55.046264 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.046223    5061 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.080169 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.080141    5061 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.080250 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.080205    5061 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.140266 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.140227    5061 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"]
Oct 30 02:40:55.140808 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.140774    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal" event={"ID":"4f18f0dafccedcaa5cd8761c40142410","Type":"ContainerStarted","Data":"52168c727bd24b83922621ea41531fa6c50fce26a97ea64b594dbb2e3b217b3b"}
Oct 30 02:40:55.140979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.140836    5061 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.140979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.140914    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal" event={"ID":"4f18f0dafccedcaa5cd8761c40142410","Type":"ContainerStarted","Data":"334191dd0df23d9916c528378d51f78f533f6ca3987f33bec6600101f20a4bd5"}
Oct 30 02:40:55.140979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.140960    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal" event={"ID":"6521a54728e46ede7b6de87a0d41821f","Type":"ContainerStarted","Data":"e4fa9b4a42b12ff66b73ed142e26c793aeea6937afad8fae41a2f68f83b12043"}
Oct 30 02:40:55.140979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.140980    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal" event={"ID":"6521a54728e46ede7b6de87a0d41821f","Type":"ContainerDied","Data":"370411047ee34b250d3138bf063e0fa9b97bd5798ad5ae523689dd7f38bdf09d"}
Oct 30 02:40:55.141183 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.140999    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal" event={"ID":"6521a54728e46ede7b6de87a0d41821f","Type":"ContainerStarted","Data":"e012fcef3d9f5dc2ef3cb13ffffb262b36d02196281a59fde68bfb10f6f4d84c"}
Oct 30 02:40:55.141183 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.141031    5061 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.164446 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.164398    5061 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:55.164446 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:55.164435    5061 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-238.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.165087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.165073    5061 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:55.165125 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:55.165106    5061 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.246323 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.246307    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.246436 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.246328    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.246436 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.246346    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f18f0dafccedcaa5cd8761c40142410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-238.ec2.internal\" (UID: \"4f18f0dafccedcaa5cd8761c40142410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.346946 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.346920    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f18f0dafccedcaa5cd8761c40142410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-238.ec2.internal\" (UID: \"4f18f0dafccedcaa5cd8761c40142410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.347035 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.346994    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.347035 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.347001    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.347035 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.347016    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f18f0dafccedcaa5cd8761c40142410-config\") pod \"kube-apiserver-proxy-ip-10-0-136-238.ec2.internal\" (UID: \"4f18f0dafccedcaa5cd8761c40142410\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.347195 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.347045    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.347195 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.347081    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6521a54728e46ede7b6de87a0d41821f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" (UID: \"6521a54728e46ede7b6de87a0d41821f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.794762 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.794729    5061 apiserver.go:52] "Watching apiserver"
Oct 30 02:40:55.842207 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.842178    5061 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:40:55.842951 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.842921    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zh9rv","openshift-network-operator/iptables-alerter-4lscq","openshift-insights/insights-runtime-extractor-kkfq8","openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5","openshift-multus/multus-dmfvd","openshift-image-registry/image-registry-67544b9897-hb2j2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal","openshift-multus/multus-additional-cni-plugins-8pbzl","openshift-dns/dns-default-m9rxl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5","openshift-ingress-canary/ingress-canary-9r4jn","openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd","openshift-service-ca/service-ca-5fffc97bdf-8t2p7","openshift-insights/periodic-gathering-q8xpw-hdnw7","openshift-image-registry/image-registry-56b7889b4-hzhbm","kube-system/konnectivity-agent-5l2pw","openshift-dns/node-resolver-qvbmn","openshift-ingress/router-default-6b959b9795-jw4rz","openshift-console-operator/console-operator-7cd56b8dbc-888jp","openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w","kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal","openshift-network-diagnostics/network-check-target-tb8xt","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2","openshift-cluster-node-tuning-operator/tuned-flkp8","openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9","openshift-image-registry/node-ca-hqwcj","openshift-multus/network-metrics-daemon-pt72w","openshift-network-console/networking-console-plugin-86c478bb88-sj78j","openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5","kube-system/kas-connection-checker-685895c7c9-cvtzh","openshift-ovn-kubernetes/ovnkube-node-vpwcs","kube-system/kas-connection-checker-685895c7c9-j86rx","kube-system/global-pull-secret-syncer-hzqcm","kube-system/kas-connection-checker-685895c7c9-7btgh","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw"]
Oct 30 02:40:55.844771 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.844752    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:55.845149 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.845131    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.845556 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.845535    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:55.847208 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.847192    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:55.849432 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.849406    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:55.857290 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.857270    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:55.859012 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.858993    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.859461 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859437    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.859556 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859439    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.859556 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859505    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.859556 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859531    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.859556 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859534    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.859794 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859565    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.859794 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859576    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.859794 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859600    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 02:40:55.859794 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859641    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.859794 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859666    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 02:40:55.859794 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859724    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 02:40:55.860086 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.859804    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 02:40:55.860278 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.860263    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:55.861426 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.861409    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 02:40:55.861572 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.861553    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 02:40:55.862018 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.861999    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 02:40:55.862092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.862057    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 02:40:55.862092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.862061    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 02:40:55.862092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.862080    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:40:55.862249 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.862118    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.862249 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.862132    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 02:40:55.862249 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.862179    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.862249 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.862239    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.862373 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.862288    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:55.862461 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.862442    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.872857 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.872817    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:40:55.872954 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.872884    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.873004 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.872947    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 02:40:55.873382 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.873366    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.873562 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.873545    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.873562 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.873558    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.873891 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.873873    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 02:40:55.873957 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.873938    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 02:40:55.874016 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.874000    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.874129 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.874002    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 02:40:55.874129 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.874054    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.874129 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.874102    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 02:40:55.874273 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.874117    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.874273 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.874161    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 02:40:55.874273 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.874259    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-prmc8\""
Oct 30 02:40:55.874364 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.874351    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 02:40:55.874486 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.874471    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gf5d6\""
Oct 30 02:40:55.877593 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.877562    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 02:40:55.877730 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.877715    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-zzvjn\""
Oct 30 02:40:55.877874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.877842    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4pml\""
Oct 30 02:40:55.878006 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.877992    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j5lkg\""
Oct 30 02:40:55.878095 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878071    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:40:55.878200 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878186    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:40:55.878279 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878260    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 02:40:55.878386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878373    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ph87b\""
Oct 30 02:40:55.878504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878490    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:40:55.878624 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878611    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qc2sc\""
Oct 30 02:40:55.878750 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878737    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:40:55.878831 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878765    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rd75s\""
Oct 30 02:40:55.878950 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878932    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.879013 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878964    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fd755\""
Oct 30 02:40:55.879013 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878964    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.879107 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878987    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:40:55.879107 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.878933    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:40:55.879107 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879081    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.879107 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879099    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:40:55.879279 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879109    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hhdzd\""
Oct 30 02:40:55.879279 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879145    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r7gc2\""
Oct 30 02:40:55.879279 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879168    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.879279 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879260    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xn2tj\""
Oct 30 02:40:55.879393 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879285    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:40:55.879393 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879316    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.879393 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879337    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5k7wg\""
Oct 30 02:40:55.879393 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879341    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:40:55.879393 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879205    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-86l9w\""
Oct 30 02:40:55.879524 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879433    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.879638 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879614    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:40:55.879695 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879084    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:40:55.879695 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879674    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.879784 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879703    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.879784 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879724    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.879784 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879736    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:40:55.879784 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879740    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.879992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879803    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.879992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879808    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.879992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879821    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:40:55.879992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879832    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 02:40:55.879992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879968    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kdxrj\""
Oct 30 02:40:55.879992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879968    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 02:40:55.879992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.879825    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:40:55.880292 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880029    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.880292 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880036    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.880292 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880072    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kk2sv\""
Oct 30 02:40:55.880292 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880167    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:40:55.880292 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880172    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xh5tx\""
Oct 30 02:40:55.880292 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880200    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.880292 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880257    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:40:55.880559 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880540    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.880613 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880568    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.880613 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880619    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:40:55.880734 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880624    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:40:55.880734 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880638    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 02:40:55.880734 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880666    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:40:55.880734 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880682    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.880734 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880701    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.880734 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880735    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mppsd\""
Oct 30 02:40:55.881008 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880755    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xsmz\""
Oct 30 02:40:55.881008 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.880788    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:40:55.903662 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.903641    5061 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.903801 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.903786    5061 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.910761 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.910743    5061 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:40:55.918636 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.918620    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 02:40:55.918754 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.918740    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 02:40:55.918786 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.918757    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 02:40:55.924642 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.924624    5061 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:55.924696 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:55.924662    5061 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-238.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.924972 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.924959    5061 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:55.925005 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:55.924990    5061 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-238.ec2.internal"
Oct 30 02:40:55.932501 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.932466    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:55.952030 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952010    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82nmv\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-kube-api-access-82nmv\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.952120 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952033    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:55.952120 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952049    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-image-registry-private-configuration\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.952120 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952069    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.952258 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952125    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-socket-dir-parent\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.952258 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952225    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrc7\" (UniqueName: \"kubernetes.io/projected/7079cfbc-daff-4782-b3ca-e5718d1d1421-kube-api-access-6qrc7\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:55.952340 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952271    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-image-registry-private-configuration\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.952340 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952274    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-netns\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.952340 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952302    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:55.952340 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952329    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16395f11-a919-4dd3-9d21-3388d472e616-config-volume\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:55.952504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952353    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-systemd-units\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.952504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952379    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-installation-pull-secrets\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.952504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952412    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-bin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.952504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952453    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:55.952504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952452    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:55.952504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952493    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-tmp-dir\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952521    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvz9z\" (UniqueName: \"kubernetes.io/projected/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-kube-api-access-zvz9z\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952536    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16395f11-a919-4dd3-9d21-3388d472e616-config-volume\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952563    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952577    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-installation-pull-secrets\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952600    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbjc\" (UniqueName: \"kubernetes.io/projected/816aaac5-8df6-402e-9367-a2ea2f8ca686-kube-api-access-6sbjc\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952610    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-tmp-dir\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952629    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-service-ca-bundle\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952655    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-kubelet\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952679    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-multus-certs\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952710    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-hosts-file\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952752    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5wx\" (UniqueName: \"kubernetes.io/projected/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-kube-api-access-6p5wx\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:55.952780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952766    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-service-ca-bundle\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.953224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952881    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-slash\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.953224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952915    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-env-overrides\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.953224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.952933    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:55.953224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953053    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-env-overrides\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.953224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953067    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-sys-fs\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:55.953224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953100    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt4f4\" (UniqueName: \"kubernetes.io/projected/16395f11-a919-4dd3-9d21-3388d472e616-kube-api-access-dt4f4\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:55.953224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953129    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:55.953224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953127    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:55.953224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953158    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-run\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.953224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953174    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.953224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953191    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953293    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-hostroot\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953319    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs586\" (UniqueName: \"kubernetes.io/projected/51189dec-b0c6-430e-9768-8f9532bdb22a-kube-api-access-cs586\") pod \"network-check-source-6f78d59f99-tngw9\" (UID: \"51189dec-b0c6-430e-9768-8f9532bdb22a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953348    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd57c8d3-5a2c-4068-b04a-901e5795b815-ca-trust-extracted\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953367    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55nbl\" (UniqueName: \"kubernetes.io/projected/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-api-access-55nbl\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953392    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-tmp\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953425    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbq6\" (UniqueName: \"kubernetes.io/projected/2967919e-d201-48ae-b388-85fe023e1d21-kube-api-access-rmbq6\") pod \"volume-data-source-validator-58c9f5f46d-cd7dw\" (UID: \"2967919e-d201-48ae-b388-85fe023e1d21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953442    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd57c8d3-5a2c-4068-b04a-901e5795b815-ca-trust-extracted\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953450    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76243024-183c-42e6-89fd-ccd82d3571c4-serving-cert\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953467    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-tmp\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953470    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953492    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-conf-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953519    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caeb5235-93eb-4b90-8372-1c59c807485f-agent-certs\") pod \"konnectivity-agent-5l2pw\" (UID: \"caeb5235-93eb-4b90-8372-1c59c807485f\") " pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953575    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw4rt\" (UniqueName: \"kubernetes.io/projected/7f9c5ebb-f865-483c-9ee3-a44c74aa3abe-kube-api-access-kw4rt\") pod \"migrator-66df5d496b-vd9k5\" (UID: \"7f9c5ebb-f865-483c-9ee3-a44c74aa3abe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953610    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-var-lib-kubelet\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953625    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76243024-183c-42e6-89fd-ccd82d3571c4-serving-cert\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.953647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953635    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953673    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-script-lib\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953699    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7cpv\" (UniqueName: \"kubernetes.io/projected/7b6386c9-d170-4b89-bb41-9a0641207188-kube-api-access-d7cpv\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953729    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-cnibin\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953755    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953780    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953840    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnvvc\" (UniqueName: \"kubernetes.io/projected/1ec92445-aa69-4865-9d4e-8a1423bf7073-kube-api-access-wnvvc\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953874    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caeb5235-93eb-4b90-8372-1c59c807485f-agent-certs\") pod \"konnectivity-agent-5l2pw\" (UID: \"caeb5235-93eb-4b90-8372-1c59c807485f\") " pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953890    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-script-lib\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953894    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzh75\" (UniqueName: \"kubernetes.io/projected/6c28bc38-2c97-412f-89b0-f7a552b78373-kube-api-access-lzh75\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953936    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-trusted-ca\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953963    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.953995    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954024    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954035    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6qd\" (UniqueName: \"kubernetes.io/projected/04ea065c-0b57-43ec-99be-989bb703b2c5-kube-api-access-dw6qd\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954129    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa5e5905-3e48-434f-8153-fd40123a614a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954162    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954240    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-registration-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:55.954259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954270    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-default-certificate\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954289    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-config\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954294    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954325    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-trusted-ca\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954337    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/960da804-13fa-4a2b-8837-1dd6f8ca83c3-iptables-alerter-script\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954396    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-trusted-ca-bundle\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954479    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b6386c9-d170-4b89-bb41-9a0641207188-config\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954509    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-default-certificate\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954521    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-stats-auth\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954647    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-system-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954650    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/960da804-13fa-4a2b-8837-1dd6f8ca83c3-iptables-alerter-script\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954702    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-image-registry-private-configuration\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954718    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-trusted-ca-bundle\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954732    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04ea065c-0b57-43ec-99be-989bb703b2c5-serving-cert\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954761    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954803    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-tuned\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954831    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16395f11-a919-4dd3-9d21-3388d472e616-tmp-dir\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954843    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-stats-auth\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954865    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-image-registry-private-configuration\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954871    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-kubernetes\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954898    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6996\" (UniqueName: \"kubernetes.io/projected/960da804-13fa-4a2b-8837-1dd6f8ca83c3-kube-api-access-t6996\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954922    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-netd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954923    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-tuned\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954932    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16395f11-a919-4dd3-9d21-3388d472e616-tmp-dir\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:55.954987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.954961    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526jr\" (UniqueName: \"kubernetes.io/projected/4844aa91-37e4-4cae-a380-af0000c768fc-kube-api-access-526jr\") pod \"kas-connection-checker-685895c7c9-7btgh\" (UID: \"4844aa91-37e4-4cae-a380-af0000c768fc\") " pod="kube-system/kas-connection-checker-685895c7c9-7btgh"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955018    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04ea065c-0b57-43ec-99be-989bb703b2c5-serving-cert\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955028    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-snapshots\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955108    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-k8s-cni-cncf-io\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955137    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/76243024-183c-42e6-89fd-ccd82d3571c4-snapshots\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955144    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-installation-pull-secrets\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955171    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrsj\" (UniqueName: \"kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj\") pod \"network-check-target-tb8xt\" (UID: \"00bc8689-1f9b-40a7-9f19-972395d1bc69\") " pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955198    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-lib-modules\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955223    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/69caed51-464d-4dab-b4a4-b4ff77da5e4e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955246    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af778722-0e75-4ddd-9819-d783cfa858e1-host\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955281    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-service-ca-bundle\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955339    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-installation-pull-secrets\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955380    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-data-volume\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955429    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/69caed51-464d-4dab-b4a4-b4ff77da5e4e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955438    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-etc-kubernetes\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955466    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-service-ca-bundle\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955465    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-data-volume\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955484    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/960da804-13fa-4a2b-8837-1dd6f8ca83c3-host-slash\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955515    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-etc-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955540    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27s6x\" (UniqueName: \"kubernetes.io/projected/cb4e2245-df10-43f4-a52a-433997bc0ca4-kube-api-access-27s6x\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955581    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-multus-daemon-config\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955659    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-cabundle\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955741    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b6386c9-d170-4b89-bb41-9a0641207188-serving-cert\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955777    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-multus-daemon-config\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.955779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955774    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955807    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955833    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfg7\" (UniqueName: \"kubernetes.io/projected/fa5e5905-3e48-434f-8153-fd40123a614a-kube-api-access-fvfg7\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955810    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-cabundle\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955877    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955929    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.955999    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-kubelet\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956001    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b6386c9-d170-4b89-bb41-9a0641207188-serving-cert\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956026    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956043    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfg6j\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-kube-api-access-dfg6j\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956162    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-socket-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956189    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-key\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956207    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-certificates\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956225    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-sys\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956248    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-tmp\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956272    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-ovn\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956315    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ec92445-aa69-4865-9d4e-8a1423bf7073-tmp\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956325    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghvzs\" (UniqueName: \"kubernetes.io/projected/bf8a2708-a7fe-41b0-8f63-7850df383c8f-kube-api-access-ghvzs\") pod \"kas-connection-checker-685895c7c9-cvtzh\" (UID: \"bf8a2708-a7fe-41b0-8f63-7850df383c8f\") " pod="kube-system/kas-connection-checker-685895c7c9-cvtzh"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956357    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kk4c\" (UniqueName: \"kubernetes.io/projected/76243024-183c-42e6-89fd-ccd82d3571c4-kube-api-access-5kk4c\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956389    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-certificates\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956395    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956422    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-var-lib-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956440    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjrl\" (UniqueName: \"kubernetes.io/projected/af778722-0e75-4ddd-9819-d783cfa858e1-kube-api-access-jjjrl\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956489    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d786d081-b91d-475a-937b-021d17c210b1-archives-path\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956556    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5wg8\" (UniqueName: \"kubernetes.io/projected/d786d081-b91d-475a-937b-021d17c210b1-kube-api-access-b5wg8\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956566    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d786d081-b91d-475a-937b-021d17c210b1-archives-path\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956560    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2bf413f-45b3-450c-b797-4e674e8da8ec-original-pull-secret\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956588    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956619    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-multus\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956648    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgnv\" (UniqueName: \"kubernetes.io/projected/1ea0be63-7705-40e8-b2a1-e48c39259c8b-kube-api-access-pvgnv\") pod \"kas-connection-checker-685895c7c9-j86rx\" (UID: \"1ea0be63-7705-40e8-b2a1-e48c39259c8b\") " pod="kube-system/kas-connection-checker-685895c7c9-j86rx"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956672    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-dbus\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:55.956685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956698    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-bound-sa-token\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956713    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-systemd\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956738    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvbf\" (UniqueName: \"kubernetes.io/projected/92515440-cc1f-4e4d-b45e-6afaff621108-kube-api-access-qnvbf\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956753    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-host\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956767    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956782    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-cni-binary-copy\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956799    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-trusted-ca\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956814    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-modprobe-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956869    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-conf\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956920    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af778722-0e75-4ddd-9819-d783cfa858e1-serviceca\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.956988    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-trusted-ca\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957030    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957035    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92515440-cc1f-4e4d-b45e-6afaff621108-cni-binary-copy\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957053    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-device-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957105    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957131    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-bin\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957137    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-trusted-ca\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957156    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-ca-trust-extracted\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957192    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-cnibin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957211    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-ca-trust-extracted\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957220    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957196    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af778722-0e75-4ddd-9819-d783cfa858e1-serviceca\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957259    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957288    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0334aa2-bc83-440a-8781-60817828e1ab-cert\") pod \"ingress-canary-9r4jn\" (UID: \"e0334aa2-bc83-440a-8781-60817828e1ab\") " pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957315    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-trusted-ca\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957326    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-systemd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957408    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjbq\" (UniqueName: \"kubernetes.io/projected/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-kube-api-access-ptjbq\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957458    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0334aa2-bc83-440a-8781-60817828e1ab-cert\") pod \"ingress-canary-9r4jn\" (UID: \"e0334aa2-bc83-440a-8781-60817828e1ab\") " pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957460    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76243024-183c-42e6-89fd-ccd82d3571c4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957546    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-config\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957575    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovn-node-metrics-cert\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957595    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-certificates\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957734    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-certificates\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957757    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysconfig\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957778    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tllq7\" (UniqueName: \"kubernetes.io/projected/e0334aa2-bc83-440a-8781-60817828e1ab-kube-api-access-tllq7\") pod \"ingress-canary-9r4jn\" (UID: \"e0334aa2-bc83-440a-8781-60817828e1ab\") " pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957804    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovn-node-metrics-cert\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957815    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caeb5235-93eb-4b90-8372-1c59c807485f-konnectivity-ca\") pod \"konnectivity-agent-5l2pw\" (UID: \"caeb5235-93eb-4b90-8372-1c59c807485f\") " pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:55.957824 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957843    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-node-log\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957888    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957916    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-bound-sa-token\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957942    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzsx\" (UniqueName: \"kubernetes.io/projected/00b0139b-56b5-4daa-8fef-b86d0ac5a1d8-kube-api-access-4fzsx\") pod \"migrator-66df5d496b-8slr5\" (UID: \"00b0139b-56b5-4daa-8fef-b86d0ac5a1d8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957978    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-kubelet-config\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.958009    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.958008    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caeb5235-93eb-4b90-8372-1c59c807485f-konnectivity-ca\") pod \"konnectivity-agent-5l2pw\" (UID: \"caeb5235-93eb-4b90-8372-1c59c807485f\") " pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.958023    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x9rq\" (UniqueName: \"kubernetes.io/projected/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-kube-api-access-9x9rq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.957988    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/816aaac5-8df6-402e-9367-a2ea2f8ca686-ovnkube-config\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.958047    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04ea065c-0b57-43ec-99be-989bb703b2c5-config\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.958104    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vgdcn\""
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.958113    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-log-socket\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.958146    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-crio-socket\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.958183    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.958215    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04ea065c-0b57-43ec-99be-989bb703b2c5-config\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.958216    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-netns\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.958952 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.958263    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qqnb\" (UniqueName: \"kubernetes.io/projected/2c04bf2f-3e60-442a-8639-10feae0a3f96-kube-api-access-6qqnb\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:55.967016 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.966997    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7079cfbc-daff-4782-b3ca-e5718d1d1421-signing-key\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:55.969456 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.969439    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82nmv\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-kube-api-access-82nmv\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.969674 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.969654    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvz9z\" (UniqueName: \"kubernetes.io/projected/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-kube-api-access-zvz9z\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:55.969708 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.969670    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrc7\" (UniqueName: \"kubernetes.io/projected/7079cfbc-daff-4782-b3ca-e5718d1d1421-kube-api-access-6qrc7\") pod \"service-ca-5fffc97bdf-8t2p7\" (UID: \"7079cfbc-daff-4782-b3ca-e5718d1d1421\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8t2p7"
Oct 30 02:40:55.969708 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.969675    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbjc\" (UniqueName: \"kubernetes.io/projected/816aaac5-8df6-402e-9367-a2ea2f8ca686-kube-api-access-6sbjc\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:55.969779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.969764    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5wx\" (UniqueName: \"kubernetes.io/projected/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-kube-api-access-6p5wx\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:55.971078 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.971021    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzh75\" (UniqueName: \"kubernetes.io/projected/6c28bc38-2c97-412f-89b0-f7a552b78373-kube-api-access-lzh75\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:55.971078 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.971038    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt4f4\" (UniqueName: \"kubernetes.io/projected/16395f11-a919-4dd3-9d21-3388d472e616-kube-api-access-dt4f4\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:55.971211 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.971141    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6qd\" (UniqueName: \"kubernetes.io/projected/04ea065c-0b57-43ec-99be-989bb703b2c5-kube-api-access-dw6qd\") pod \"service-ca-operator-544d749c8c-k5t9w\" (UID: \"04ea065c-0b57-43ec-99be-989bb703b2c5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w"
Oct 30 02:40:55.971377 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.971362    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7cpv\" (UniqueName: \"kubernetes.io/projected/7b6386c9-d170-4b89-bb41-9a0641207188-kube-api-access-d7cpv\") pod \"console-operator-7cd56b8dbc-888jp\" (UID: \"7b6386c9-d170-4b89-bb41-9a0641207188\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:55.971619 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.971596    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnvvc\" (UniqueName: \"kubernetes.io/projected/1ec92445-aa69-4865-9d4e-8a1423bf7073-kube-api-access-wnvvc\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:55.971674 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.971645    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55nbl\" (UniqueName: \"kubernetes.io/projected/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-api-access-55nbl\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:55.971721 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.971690    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs586\" (UniqueName: \"kubernetes.io/projected/51189dec-b0c6-430e-9768-8f9532bdb22a-kube-api-access-cs586\") pod \"network-check-source-6f78d59f99-tngw9\" (UID: \"51189dec-b0c6-430e-9768-8f9532bdb22a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tngw9"
Oct 30 02:40:55.971958 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.971943    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw4rt\" (UniqueName: \"kubernetes.io/projected/7f9c5ebb-f865-483c-9ee3-a44c74aa3abe-kube-api-access-kw4rt\") pod \"migrator-66df5d496b-vd9k5\" (UID: \"7f9c5ebb-f865-483c-9ee3-a44c74aa3abe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd9k5"
Oct 30 02:40:55.972010 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.971994    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbq6\" (UniqueName: \"kubernetes.io/projected/2967919e-d201-48ae-b388-85fe023e1d21-kube-api-access-rmbq6\") pod \"volume-data-source-validator-58c9f5f46d-cd7dw\" (UID: \"2967919e-d201-48ae-b388-85fe023e1d21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cd7dw"
Oct 30 02:40:55.972249 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.972228    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6996\" (UniqueName: \"kubernetes.io/projected/960da804-13fa-4a2b-8837-1dd6f8ca83c3-kube-api-access-t6996\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:55.972469 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.972455    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzrsj\" (UniqueName: \"kubernetes.io/projected/00bc8689-1f9b-40a7-9f19-972395d1bc69-kube-api-access-wzrsj\") pod \"network-check-target-tb8xt\" (UID: \"00bc8689-1f9b-40a7-9f19-972395d1bc69\") " pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:55.973691 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.973675    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-526jr\" (UniqueName: \"kubernetes.io/projected/4844aa91-37e4-4cae-a380-af0000c768fc-kube-api-access-526jr\") pod \"kas-connection-checker-685895c7c9-7btgh\" (UID: \"4844aa91-37e4-4cae-a380-af0000c768fc\") " pod="kube-system/kas-connection-checker-685895c7c9-7btgh"
Oct 30 02:40:55.973795 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.973779    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27s6x\" (UniqueName: \"kubernetes.io/projected/cb4e2245-df10-43f4-a52a-433997bc0ca4-kube-api-access-27s6x\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:55.975162 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.975142    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfg6j\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-kube-api-access-dfg6j\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.975209 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.975163    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kk4c\" (UniqueName: \"kubernetes.io/projected/76243024-183c-42e6-89fd-ccd82d3571c4-kube-api-access-5kk4c\") pod \"insights-operator-7c4986d449-zh9rv\" (UID: \"76243024-183c-42e6-89fd-ccd82d3571c4\") " pod="openshift-insights/insights-operator-7c4986d449-zh9rv"
Oct 30 02:40:55.975209 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.975146    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfg7\" (UniqueName: \"kubernetes.io/projected/fa5e5905-3e48-434f-8153-fd40123a614a-kube-api-access-fvfg7\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:55.976195 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.976154    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-bound-sa-token\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:55.976288 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.976274    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjbq\" (UniqueName: \"kubernetes.io/projected/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-kube-api-access-ptjbq\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:55.976339 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.976323    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x9rq\" (UniqueName: \"kubernetes.io/projected/47e2d60c-0e3c-4c87-9198-b2d5bc0122c8-kube-api-access-9x9rq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2\" (UID: \"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2"
Oct 30 02:40:55.977009 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.976984    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qqnb\" (UniqueName: \"kubernetes.io/projected/2c04bf2f-3e60-442a-8639-10feae0a3f96-kube-api-access-6qqnb\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:55.977107 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.977047    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-bound-sa-token\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:55.977169 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.977113    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzsx\" (UniqueName: \"kubernetes.io/projected/00b0139b-56b5-4daa-8fef-b86d0ac5a1d8-kube-api-access-4fzsx\") pod \"migrator-66df5d496b-8slr5\" (UID: \"00b0139b-56b5-4daa-8fef-b86d0ac5a1d8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8slr5"
Oct 30 02:40:55.977237 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.977216    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvbf\" (UniqueName: \"kubernetes.io/projected/92515440-cc1f-4e4d-b45e-6afaff621108-kube-api-access-qnvbf\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:55.977278 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.977241    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5wg8\" (UniqueName: \"kubernetes.io/projected/d786d081-b91d-475a-937b-021d17c210b1-kube-api-access-b5wg8\") pod \"periodic-gathering-q8xpw-hdnw7\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") " pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:40:55.977348 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.977330    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tllq7\" (UniqueName: \"kubernetes.io/projected/e0334aa2-bc83-440a-8781-60817828e1ab-kube-api-access-tllq7\") pod \"ingress-canary-9r4jn\" (UID: \"e0334aa2-bc83-440a-8781-60817828e1ab\") " pod="openshift-ingress-canary/ingress-canary-9r4jn"
Oct 30 02:40:55.977791 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.977772    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgnv\" (UniqueName: \"kubernetes.io/projected/1ea0be63-7705-40e8-b2a1-e48c39259c8b-kube-api-access-pvgnv\") pod \"kas-connection-checker-685895c7c9-j86rx\" (UID: \"1ea0be63-7705-40e8-b2a1-e48c39259c8b\") " pod="kube-system/kas-connection-checker-685895c7c9-j86rx"
Oct 30 02:40:55.977915 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.977888    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjrl\" (UniqueName: \"kubernetes.io/projected/af778722-0e75-4ddd-9819-d783cfa858e1-kube-api-access-jjjrl\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:55.978561 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:55.978542    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghvzs\" (UniqueName: \"kubernetes.io/projected/bf8a2708-a7fe-41b0-8f63-7850df383c8f-kube-api-access-ghvzs\") pod \"kas-connection-checker-685895c7c9-cvtzh\" (UID: \"bf8a2708-a7fe-41b0-8f63-7850df383c8f\") " pod="kube-system/kas-connection-checker-685895c7c9-cvtzh"
Oct 30 02:40:56.059243 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059164    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-sys\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.059243 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059205    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-ovn\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.059243 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059230    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-var-lib-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059266    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059284    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-sys\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059292    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-multus\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059316    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-dbus\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059321    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-var-lib-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059335    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-multus\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059292    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-ovn\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059358    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-systemd\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059430    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-host\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059452    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-systemd\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059463    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059475    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-dbus\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:56.059484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059486    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-modprobe-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059509    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-conf\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059534    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059548    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-device-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059570    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-modprobe-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059478    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-host\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059606    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-bin\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059621    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059637    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-cnibin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059657    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-conf\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059613    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-device-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059642    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-bin\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059686    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059694    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-cnibin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059720    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059744    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-systemd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059769    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysconfig\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059793    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-node-log\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059817    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059857    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-kubelet-config\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059867    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-node-log\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059876    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysconfig\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059796    5061 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059890    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-log-socket\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059914    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-crio-socket\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059916    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059924    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2bf413f-45b3-450c-b797-4e674e8da8ec-kubelet-config\") pod \"global-pull-secret-syncer-hzqcm\" (UID: \"a2bf413f-45b3-450c-b797-4e674e8da8ec\") " pod="kube-system/global-pull-secret-syncer-hzqcm"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059938    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059951    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059967    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-log-socket\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059971    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-netns\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.059817    5061 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060005    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060003    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-netns\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060039    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-socket-dir-parent\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.060049    5061 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.060057    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs podName:2c04bf2f-3e60-442a-8639-10feae0a3f96 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:56.56003506 +0000 UTC m=+1.862213435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs") pod "network-metrics-daemon-pt72w" (UID: "2c04bf2f-3e60-442a-8639-10feae0a3f96") : secret "metrics-daemon-secret" not found
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060080    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-socket-dir-parent\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060006    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-crio-socket\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060088    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-netns\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060079    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060116    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:56.060112 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.060143    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:56.560125503 +0000 UTC m=+1.862303846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060148    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-netns\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060172    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-systemd-units\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.060181    5061 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060200    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-bin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.060217    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:56.560205471 +0000 UTC m=+1.862383850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : secret "router-metrics-certs-default" not found
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060246    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-cni-bin\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060243    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.059883    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-systemd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060283    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-systemd-units\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060294    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060290    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060333    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060365    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-kubelet\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060389    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-multus-certs\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060411    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-hosts-file\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060433    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-slash\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060458    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-sys-fs\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060462    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-multus-certs\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060483    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-run\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060484    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d7330ef-3e60-407f-89ca-0c0c9e38c87d-hosts-file\") pod \"node-resolver-qvbmn\" (UID: \"5d7330ef-3e60-407f-89ca-0c0c9e38c87d\") " pod="openshift-dns/node-resolver-qvbmn"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060510    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060520    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-sys-fs\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060532    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-run\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060573    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060603    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-hostroot\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060634    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-conf-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060661    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-var-lib-kubelet\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060671    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-multus-conf-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060636    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-hostroot\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060687    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.060718    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:56.560699508 +0000 UTC m=+1.862877859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : configmap references non-existent config key: service-ca.crt
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060741    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-var-lib-kubelet\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.060783    5061 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060789    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-cnibin\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060838    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060893    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-slash\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060901    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-cnibin\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060902    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.060923    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls podName:c9aa32ac-0748-4d2f-820d-8a15ec78cdb1 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:56.560911208 +0000 UTC m=+1.863089554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gl8sd" (UID: "c9aa32ac-0748-4d2f-820d-8a15ec78cdb1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060930    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa5e5905-3e48-434f-8153-fd40123a614a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbzl\" (UID: \"fa5e5905-3e48-434f-8153-fd40123a614a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbzl"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060971    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-registration-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.060974    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-var-lib-kubelet\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061007    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-system-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061049    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061073    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-kubernetes\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061078    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061089    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-system-cni-dir\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061112    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-netd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061129    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-registration-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061135    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-kubernetes\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061138    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-k8s-cni-cncf-io\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061164    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-lib-modules\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061195    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-host-run-k8s-cni-cncf-io\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061202    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af778722-0e75-4ddd-9819-d783cfa858e1-host\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061228    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-etc-kubernetes\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061231    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-cni-netd\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061263    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/960da804-13fa-4a2b-8837-1dd6f8ca83c3-host-slash\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061286    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-lib-modules\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061290    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-etc-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061301    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92515440-cc1f-4e4d-b45e-6afaff621108-etc-kubernetes\") pod \"multus-dmfvd\" (UID: \"92515440-cc1f-4e4d-b45e-6afaff621108\") " pod="openshift-multus/multus-dmfvd"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061331    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061325    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-etc-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061354    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/960da804-13fa-4a2b-8837-1dd6f8ca83c3-host-slash\") pod \"iptables-alerter-4lscq\" (UID: \"960da804-13fa-4a2b-8837-1dd6f8ca83c3\") " pod="openshift-network-operator/iptables-alerter-4lscq"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061355    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061386    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-run-openvswitch\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061393    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061422    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-kubelet\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061448    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-socket-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061531    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb4e2245-df10-43f4-a52a-433997bc0ca4-socket-dir\") pod \"aws-ebs-csi-driver-node-pvtl5\" (UID: \"cb4e2245-df10-43f4-a52a-433997bc0ca4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvtl5"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061534    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ec92445-aa69-4865-9d4e-8a1423bf7073-etc-sysctl-d\") pod \"tuned-flkp8\" (UID: \"1ec92445-aa69-4865-9d4e-8a1423bf7073\") " pod="openshift-cluster-node-tuning-operator/tuned-flkp8"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061567    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af778722-0e75-4ddd-9819-d783cfa858e1-host\") pod \"node-ca-hqwcj\" (UID: \"af778722-0e75-4ddd-9819-d783cfa858e1\") " pod="openshift-image-registry/node-ca-hqwcj"
Oct 30 02:40:56.061890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.061606    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/816aaac5-8df6-402e-9367-a2ea2f8ca686-host-kubelet\") pod \"ovnkube-node-vpwcs\" (UID: \"816aaac5-8df6-402e-9367-a2ea2f8ca686\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:56.065172 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.062515    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395f11-a919-4dd3-9d21-3388d472e616-metrics-tls\") pod \"dns-default-m9rxl\" (UID: \"16395f11-a919-4dd3-9d21-3388d472e616\") " pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:56.065172 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.062535    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/215f867e-a2ba-4b0d-ada3-6182c5e1a8fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pppd8\" (UID: \"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:56.065172 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.063436    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69caed51-464d-4dab-b4a4-b4ff77da5e4e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sj78j\" (UID: \"69caed51-464d-4dab-b4a4-b4ff77da5e4e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:56.065172 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.063829    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\") pod \"image-registry-56b7889b4-hzhbm\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") " pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:56.065172 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.064357    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\") pod \"image-registry-67544b9897-hb2j2\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") " pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:56.124508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.124477    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:56.145030 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.145007    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"
Oct 30 02:40:56.146090 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.146075    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:56.147723 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.147696    5061 scope.go:122] "RemoveContainer" containerID="b964cb6f1289891b21a2f16154013500efe4d5530fd822329a568441f26a926f"
Oct 30 02:40:56.149755 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.149734    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j"
Oct 30 02:40:56.159954 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.159922    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:40:56.161046 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.160997    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:56.359815 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.359754    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:40:56.485298 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.485259    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8"]
Oct 30 02:40:56.489069 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.488802    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67544b9897-hb2j2"]
Oct 30 02:40:56.494308 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:56.494281    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd57c8d3_5a2c_4068_b04a_901e5795b815.slice/crio-890cddd7c633779ba5e74157fe9f6da640185b0cc116ff52e9ae965c9285bc11 WatchSource:0}: Error finding container 890cddd7c633779ba5e74157fe9f6da640185b0cc116ff52e9ae965c9285bc11: Status 404 returned error can't find the container with id 890cddd7c633779ba5e74157fe9f6da640185b0cc116ff52e9ae965c9285bc11
Oct 30 02:40:56.523762 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.523703    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sj78j"]
Oct 30 02:40:56.525389 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.525373    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:40:56.527684 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.527660    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56b7889b4-hzhbm"]
Oct 30 02:40:56.529442 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:56.529420    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69caed51_464d_4dab_b4a4_b4ff77da5e4e.slice/crio-b94c88c437a7ef9295f8437fa55098e1b4740c676bf85e77c0f63e43c28db279 WatchSource:0}: Error finding container b94c88c437a7ef9295f8437fa55098e1b4740c676bf85e77c0f63e43c28db279: Status 404 returned error can't find the container with id b94c88c437a7ef9295f8437fa55098e1b4740c676bf85e77c0f63e43c28db279
Oct 30 02:40:56.529555 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.529536    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m9rxl"]
Oct 30 02:40:56.533717 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:56.533694    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8a405e_f80a_4aa9_ab4c_2217e9250b70.slice/crio-a41a695233fece2f22d5fd0da949b94c813bb3e8613e1ef49a0fd744c4dac412 WatchSource:0}: Error finding container a41a695233fece2f22d5fd0da949b94c813bb3e8613e1ef49a0fd744c4dac412: Status 404 returned error can't find the container with id a41a695233fece2f22d5fd0da949b94c813bb3e8613e1ef49a0fd744c4dac412
Oct 30 02:40:56.535598 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:40:56.535575    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16395f11_a919_4dd3_9d21_3388d472e616.slice/crio-22780ed8392b145c16f49832c176bfb3a5d25327c52fcc355bd9dc0e2c6377d1 WatchSource:0}: Error finding container 22780ed8392b145c16f49832c176bfb3a5d25327c52fcc355bd9dc0e2c6377d1: Status 404 returned error can't find the container with id 22780ed8392b145c16f49832c176bfb3a5d25327c52fcc355bd9dc0e2c6377d1
Oct 30 02:40:56.565583 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.565555    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:56.565686 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.565590    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:56.565686 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.565623    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:56.565686 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.565652    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:56.565819 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.565711    5061 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:56.565819 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.565743    5061 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:40:56.565819 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.565757    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:57.565739075 +0000 UTC m=+2.867917421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : configmap references non-existent config key: service-ca.crt
Oct 30 02:40:56.565819 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.565775    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:57.565766388 +0000 UTC m=+2.867944732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:56.565819 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.565788    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:57.565781284 +0000 UTC m=+2.867959630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : secret "router-metrics-certs-default" not found
Oct 30 02:40:56.565819 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.565802    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:56.566052 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.565905    5061 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:56.566052 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:56.565942    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls podName:c9aa32ac-0748-4d2f-820d-8a15ec78cdb1 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:57.565930084 +0000 UTC m=+2.868108442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gl8sd" (UID: "c9aa32ac-0748-4d2f-820d-8a15ec78cdb1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:56.567728 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.567707    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c04bf2f-3e60-442a-8639-10feae0a3f96-metrics-certs\") pod \"network-metrics-daemon-pt72w\" (UID: \"2c04bf2f-3e60-442a-8639-10feae0a3f96\") " pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:56.732136 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.732060    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:56.758220 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.758191    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pt72w"
Oct 30 02:40:56.910200 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.910130    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m9rxl" event={"ID":"16395f11-a919-4dd3-9d21-3388d472e616","Type":"ContainerStarted","Data":"22780ed8392b145c16f49832c176bfb3a5d25327c52fcc355bd9dc0e2c6377d1"}
Oct 30 02:40:56.915260 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.914515    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67544b9897-hb2j2" event={"ID":"cd57c8d3-5a2c-4068-b04a-901e5795b815","Type":"ContainerStarted","Data":"84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30"}
Oct 30 02:40:56.915260 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.914572    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67544b9897-hb2j2" event={"ID":"cd57c8d3-5a2c-4068-b04a-901e5795b815","Type":"ContainerStarted","Data":"890cddd7c633779ba5e74157fe9f6da640185b0cc116ff52e9ae965c9285bc11"}
Oct 30 02:40:56.917429 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.917398    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm" event={"ID":"4f8a405e-f80a-4aa9-ab4c-2217e9250b70","Type":"ContainerStarted","Data":"7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8"}
Oct 30 02:40:56.917532 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.917446    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm" event={"ID":"4f8a405e-f80a-4aa9-ab4c-2217e9250b70","Type":"ContainerStarted","Data":"a41a695233fece2f22d5fd0da949b94c813bb3e8613e1ef49a0fd744c4dac412"}
Oct 30 02:40:56.921022 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.920440    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:40:56.921022 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.920535    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp" event={"ID":"7b6386c9-d170-4b89-bb41-9a0641207188","Type":"ContainerStarted","Data":"e0e52191ef135ec885c42282bbccca5f9aaca660517147ce55ee4321cc79278e"}
Oct 30 02:40:56.926392 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.926315    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j" event={"ID":"69caed51-464d-4dab-b4a4-b4ff77da5e4e","Type":"ContainerStarted","Data":"b94c88c437a7ef9295f8437fa55098e1b4740c676bf85e77c0f63e43c28db279"}
Oct 30 02:40:56.931091 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:56.930230    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8" event={"ID":"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd","Type":"ContainerStarted","Data":"e78929e81910d7594ba9cbd5e080d23dc3e8b764c8fa81d91b34d4021fae5ccd"}
Oct 30 02:40:57.029307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.029233    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:57.077879 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.069236    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pt72w"]
Oct 30 02:40:57.098592 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.098383    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:57.104879 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.104658    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-888jp"
Oct 30 02:40:57.146368 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.146335    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:57.261513 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.261479    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:57.421812 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.421743    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:57.546094 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.546026    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:57.579583 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.578534    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:57.579583 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.578586    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:57.579583 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.578629    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:57.579583 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.578663    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:57.579583 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:57.578806    5061 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:57.579583 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:57.578885    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls podName:c9aa32ac-0748-4d2f-820d-8a15ec78cdb1 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:59.578867254 +0000 UTC m=+4.881045610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gl8sd" (UID: "c9aa32ac-0748-4d2f-820d-8a15ec78cdb1") : secret "cluster-monitoring-operator-tls" not found
Oct 30 02:40:57.579583 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:57.579350    5061 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:57.579583 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:57.579407    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:40:59.579390021 +0000 UTC m=+4.881568378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:57.579583 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:57.579462    5061 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:40:57.579583 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:57.579492    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:59.579481737 +0000 UTC m=+4.881660094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : secret "router-metrics-certs-default" not found
Oct 30 02:40:57.579583 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:57.579550    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:59.579540715 +0000 UTC m=+4.881719072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : configmap references non-existent config key: service-ca.crt
Oct 30 02:40:57.677665 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.677544    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:57.794660 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.794629    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:57.932876 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.932756    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:57.936617 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.936584    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-sj78j" event={"ID":"69caed51-464d-4dab-b4a4-b4ff77da5e4e","Type":"ContainerStarted","Data":"b83da54202c16748b39fb1994836a6f016c17334b93c42c0c1829037775f59b6"}
Oct 30 02:40:57.938298 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.938277    5061 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:40:57.938713 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.938696    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pt72w" event={"ID":"2c04bf2f-3e60-442a-8639-10feae0a3f96","Type":"ContainerStarted","Data":"fc1006a2d8b2c5009728fbc714dde0cb4629e5526e4561371f4ee27888e237d2"}
Oct 30 02:40:57.939151 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.939136    5061 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:40:57.943720 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:57.943703    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:40:58.337164 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:58.337140    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:58.796433 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:58.796406    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:58.944379 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:58.944338    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m9rxl" event={"ID":"16395f11-a919-4dd3-9d21-3388d472e616","Type":"ContainerStarted","Data":"0b20091dc1c09740f505da413f6114c9a5ca2382d78f2b28d39805c10d7e4eb3"}
Oct 30 02:40:58.944780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:58.944409    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m9rxl" event={"ID":"16395f11-a919-4dd3-9d21-3388d472e616","Type":"ContainerStarted","Data":"8298112fa48e4163b7c60f3b5394a40b7c76e75a0d91e9fdbe2fb1509623bb01"}
Oct 30 02:40:58.946184 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:58.946157    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8" event={"ID":"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd","Type":"ContainerStarted","Data":"54244cb295357c6165aac1349750ad49fb23360ce996e4ecfc090b24334321c9"}
Oct 30 02:40:58.946242 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:58.946214    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pppd8" event={"ID":"215f867e-a2ba-4b0d-ada3-6182c5e1a8fd","Type":"ContainerStarted","Data":"88d54ba9103a2aa90c86d9ea8e25d00cb3919c0446d1f0a52a6f3a2461a67768"}
Oct 30 02:40:58.950571 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:58.950546    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pt72w" event={"ID":"2c04bf2f-3e60-442a-8639-10feae0a3f96","Type":"ContainerStarted","Data":"7643e8dcbc070a1db3b9b97ffafd4d9080149d822f45539589c4b65ddd6b452e"}
Oct 30 02:40:58.950692 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:58.950590    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pt72w" event={"ID":"2c04bf2f-3e60-442a-8639-10feae0a3f96","Type":"ContainerStarted","Data":"1ed10a51a056d14cb67403056e4c14d6a4e6c00dd8b61784937faf1eb4c76a6c"}
Oct 30 02:40:58.953220 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:58.953202    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:40:59.143819 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.143788    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:59.531154 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.531088    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:59.598345 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.598312    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:40:59.598501 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.598353    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:59.598501 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.598393    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:59.598501 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.598416    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:59.598501 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:59.598460    5061 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:59.598501 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:59.598494    5061 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 02:40:59.598700 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:59.598549    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs podName:6c28bc38-2c97-412f-89b0-f7a552b78373 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:03.598531238 +0000 UTC m=+8.900709595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs") pod "router-default-6b959b9795-jw4rz" (UID: "6c28bc38-2c97-412f-89b0-f7a552b78373") : secret "router-metrics-certs-default" not found
Oct 30 02:40:59.598700 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:40:59.598563    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls podName:33f38b3e-2ca5-40e1-ab36-f175b4a905fd nodeName:}" failed. No retries permitted until 2026-10-30 02:41:03.598557385 +0000 UTC m=+8.900735728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkfq8" (UID: "33f38b3e-2ca5-40e1-ab36-f175b4a905fd") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:59.599153 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.599132    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c28bc38-2c97-412f-89b0-f7a552b78373-service-ca-bundle\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:40:59.600948 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.600926    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9aa32ac-0748-4d2f-820d-8a15ec78cdb1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gl8sd\" (UID: \"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:59.652716 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.652631    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:59.748341 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.748309    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"
Oct 30 02:40:59.786156 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.786090    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5l2pw"
Oct 30 02:40:59.823563 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.823532    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:59.926240 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.926220    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:40:59.941510 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.941490    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:59.958574 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.958548    5061 generic.go:356] "Generic (PLEG): container finished" podID="04ea065c-0b57-43ec-99be-989bb703b2c5" containerID="5f11d73b2bd4ead981af1cc4d6c8e267a126fb29c3fe412efc08c9188ce1ad54" exitCode=0
Oct 30 02:40:59.958909 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.958593    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w" event={"ID":"04ea065c-0b57-43ec-99be-989bb703b2c5","Type":"ContainerDied","Data":"5f11d73b2bd4ead981af1cc4d6c8e267a126fb29c3fe412efc08c9188ce1ad54"}
Oct 30 02:40:59.958909 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.958900    5061 scope.go:122] "RemoveContainer" containerID="5f11d73b2bd4ead981af1cc4d6c8e267a126fb29c3fe412efc08c9188ce1ad54"
Oct 30 02:40:59.960186 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.960124    5061 generic.go:356] "Generic (PLEG): container finished" podID="47e2d60c-0e3c-4c87-9198-b2d5bc0122c8" containerID="31b7a01f469f78fcf5169fbc266f77b0a2d489383c5edbfce77fdf772b26cffb" exitCode=0
Oct 30 02:40:59.960250 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.960207    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2" event={"ID":"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8","Type":"ContainerDied","Data":"31b7a01f469f78fcf5169fbc266f77b0a2d489383c5edbfce77fdf772b26cffb"}
Oct 30 02:40:59.960868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:40:59.960838    5061 scope.go:122] "RemoveContainer" containerID="31b7a01f469f78fcf5169fbc266f77b0a2d489383c5edbfce77fdf772b26cffb"
Oct 30 02:41:00.045181 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:00.045113    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd"]
Oct 30 02:41:00.308679 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:00.308598    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:41:00.403900 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:00.403862    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:00.730797 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:00.730715    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:00.814562 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:00.814536    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:41:00.814756 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:00.814740    5061 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:00.965135 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:00.965098    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd" event={"ID":"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1","Type":"ContainerStarted","Data":"bb0aafa78f4429c6946f6e809b04fc731b48863fadfc8bb336376cd4d183a108"}
Oct 30 02:41:00.967303 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:00.967277    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-k5t9w" event={"ID":"04ea065c-0b57-43ec-99be-989bb703b2c5","Type":"ContainerStarted","Data":"1cc83bb827010c61e901494829179092571137b3fe06a4f8c5a35bed30e12732"}
Oct 30 02:41:00.969369 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:00.969346    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtrz2" event={"ID":"47e2d60c-0e3c-4c87-9198-b2d5bc0122c8","Type":"ContainerStarted","Data":"35070a2808dfb1c95b7c08f5444b1efd42d3477a8c2aba55128026f81d5c201f"}
Oct 30 02:41:01.132091 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:01.132050    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:01.542310 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:01.542283    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:01.975068 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:01.975034    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gl8sd" event={"ID":"c9aa32ac-0748-4d2f-820d-8a15ec78cdb1","Type":"ContainerStarted","Data":"9a7d9673a00f1d689bf616271c0facb78910e5d8086fdd9527ebcf753165aaf6"}
Oct 30 02:41:01.983510 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:01.983480    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:02.182453 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:02.182415    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:02.380442 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:02.380410    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:02.502670 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:02.502637    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:02.622480 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:02.622446    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:02.743554 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:02.743472    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:02.859393 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:02.859365    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:03.002873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.002766    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:03.356389 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.356354    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:03.526940 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.526912    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:03.635433 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.635339    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:41:03.635433 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.635387    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:41:03.637723 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.637695    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33f38b3e-2ca5-40e1-ab36-f175b4a905fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkfq8\" (UID: \"33f38b3e-2ca5-40e1-ab36-f175b4a905fd\") " pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:41:03.637958 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.637940    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c28bc38-2c97-412f-89b0-f7a552b78373-metrics-certs\") pod \"router-default-6b959b9795-jw4rz\" (UID: \"6c28bc38-2c97-412f-89b0-f7a552b78373\") " pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:41:03.646813 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.646780    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:41:03.662974 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.662945    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkfq8"
Oct 30 02:41:03.971086 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.971018    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kkfq8"]
Oct 30 02:41:03.976315 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:41:03.976292    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f38b3e_2ca5_40e1_ab36_f175b4a905fd.slice/crio-1c86a01d0999a3ff978b0bc63c567e0cfcaceed6a00a7beba82bbafb1c8f59dc WatchSource:0}: Error finding container 1c86a01d0999a3ff978b0bc63c567e0cfcaceed6a00a7beba82bbafb1c8f59dc: Status 404 returned error can't find the container with id 1c86a01d0999a3ff978b0bc63c567e0cfcaceed6a00a7beba82bbafb1c8f59dc
Oct 30 02:41:03.982002 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.981981    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b959b9795-jw4rz"]
Oct 30 02:41:03.984676 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:03.984641    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkfq8" event={"ID":"33f38b3e-2ca5-40e1-ab36-f175b4a905fd","Type":"ContainerStarted","Data":"1c86a01d0999a3ff978b0bc63c567e0cfcaceed6a00a7beba82bbafb1c8f59dc"}
Oct 30 02:41:03.988398 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:41:03.988376    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c28bc38_2c97_412f_89b0_f7a552b78373.slice/crio-8c58fb9deb654773509e931088b26c87e90747a27608e0574b51308218541a15 WatchSource:0}: Error finding container 8c58fb9deb654773509e931088b26c87e90747a27608e0574b51308218541a15: Status 404 returned error can't find the container with id 8c58fb9deb654773509e931088b26c87e90747a27608e0574b51308218541a15
Oct 30 02:41:04.990777 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:04.990728    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b959b9795-jw4rz" event={"ID":"6c28bc38-2c97-412f-89b0-f7a552b78373","Type":"ContainerStarted","Data":"4ef2d3c70f9df03a1d1eab94fe9d05077f582d678195cb274a0be3ade9892e30"}
Oct 30 02:41:04.991317 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:04.990797    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b959b9795-jw4rz" event={"ID":"6c28bc38-2c97-412f-89b0-f7a552b78373","Type":"ContainerStarted","Data":"8c58fb9deb654773509e931088b26c87e90747a27608e0574b51308218541a15"}
Oct 30 02:41:04.993372 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:04.993339    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkfq8" event={"ID":"33f38b3e-2ca5-40e1-ab36-f175b4a905fd","Type":"ContainerStarted","Data":"638b9d3cc988d70708d1b3a3e06725fdbea2212cc797d0ec786034e394bb470f"}
Oct 30 02:41:04.993483 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:04.993393    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkfq8" event={"ID":"33f38b3e-2ca5-40e1-ab36-f175b4a905fd","Type":"ContainerStarted","Data":"b3341955d0787812bbe83ea86aca1fe0805755c28b7388625a6ac922dfabec28"}
Oct 30 02:41:05.647245 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:05.647204    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:41:05.650366 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:05.650339    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:41:06.000008 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:05.999938    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkfq8" event={"ID":"33f38b3e-2ca5-40e1-ab36-f175b4a905fd","Type":"ContainerStarted","Data":"b55970940900265c05b29ac484c72fda499ede343ab10afa2bcce86e8ae8b2c1"}
Oct 30 02:41:06.000342 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:06.000323    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:41:06.001382 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:06.001362    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6b959b9795-jw4rz"
Oct 30 02:41:07.017058 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:07.017026    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:07.132243 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:07.132212    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:08.415605 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:08.415576    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpwcs"
Oct 30 02:41:09.972204 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:09.972169    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m9rxl"
Oct 30 02:41:12.354814 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:12.354780    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67544b9897-hb2j2"]
Oct 30 02:41:12.360286 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:12.360256    5061 patch_prober.go:29] interesting pod/image-registry-67544b9897-hb2j2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 02:41:12.360423 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:12.360310    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67544b9897-hb2j2" podUID="cd57c8d3-5a2c-4068-b04a-901e5795b815" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 02:41:12.455086 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:12.455037    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:12.588324 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:12.588289    5061 patch_prober.go:29] interesting pod/image-registry-56b7889b4-hzhbm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 02:41:12.588486 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:12.588340    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm" podUID="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 02:41:12.600744 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:12.600714    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:12.798657 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:12.798621    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:14.826527 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:14.826496    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:14.936551 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:14.936515    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:15.545674 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:15.545641    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:15.677838 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:15.677805    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:15.829899 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:15.829814    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:15.990549 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:15.990499    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:16.107350 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:16.107274    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:18.089587 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:18.089554    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:19.366107 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:19.366074    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:19.498524 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:19.498497    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:20.020761 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.020731    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:20.221310 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.221278    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:20.313690 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.313659    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:41:20.343910 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.343878    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b4fjt"]
Oct 30 02:41:20.349166 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.349147    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.365971 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.365944    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-tls\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.366084 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.365998    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3d6304c-30e6-4f4f-9327-3c4db4beac53-sys\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.366084 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.366021    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-textfile\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.366453 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.366090    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.366453 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.366126    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm8zm\" (UniqueName: \"kubernetes.io/projected/f3d6304c-30e6-4f4f-9327-3c4db4beac53-kube-api-access-dm8zm\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.366453 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.366172    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3d6304c-30e6-4f4f-9327-3c4db4beac53-root\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.366453 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.366197    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-wtmp\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.366453 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.366251    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d6304c-30e6-4f4f-9327-3c4db4beac53-metrics-client-ca\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.366453 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.366276    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-accelerators-collector-config\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.405607 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.405577    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vvql5\""
Oct 30 02:41:20.405741 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.405637    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 02:41:20.405741 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.405637    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 02:41:20.439527 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.439500    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 02:41:20.439640 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.439529    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 02:41:20.466955 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.466928    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-tls\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.466971    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3d6304c-30e6-4f4f-9327-3c4db4beac53-sys\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.466997    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-textfile\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467033    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467052    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3d6304c-30e6-4f4f-9327-3c4db4beac53-sys\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467062    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm8zm\" (UniqueName: \"kubernetes.io/projected/f3d6304c-30e6-4f4f-9327-3c4db4beac53-kube-api-access-dm8zm\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467130    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3d6304c-30e6-4f4f-9327-3c4db4beac53-root\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467156    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-wtmp\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467192    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d6304c-30e6-4f4f-9327-3c4db4beac53-metrics-client-ca\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467231    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3d6304c-30e6-4f4f-9327-3c4db4beac53-root\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467282    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-accelerators-collector-config\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467361    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-wtmp\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467697 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467412    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-textfile\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467831 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467812    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-accelerators-collector-config\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.467919 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.467815    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d6304c-30e6-4f4f-9327-3c4db4beac53-metrics-client-ca\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.469311 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.469285    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.469405 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.469392    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3d6304c-30e6-4f4f-9327-3c4db4beac53-node-exporter-tls\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.499452 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.499420    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm8zm\" (UniqueName: \"kubernetes.io/projected/f3d6304c-30e6-4f4f-9327-3c4db4beac53-kube-api-access-dm8zm\") pod \"node-exporter-b4fjt\" (UID: \"f3d6304c-30e6-4f4f-9327-3c4db4beac53\") " pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.513286 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.513258    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:20.665757 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.665701    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b4fjt"
Oct 30 02:41:20.678048 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:41:20.678024    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d6304c_30e6_4f4f_9327_3c4db4beac53.slice/crio-4ffd56045ff9135a4fed740336bb512deb6c0605fa48f246a99a6bdd3f4f3a23 WatchSource:0}: Error finding container 4ffd56045ff9135a4fed740336bb512deb6c0605fa48f246a99a6bdd3f4f3a23: Status 404 returned error can't find the container with id 4ffd56045ff9135a4fed740336bb512deb6c0605fa48f246a99a6bdd3f4f3a23
Oct 30 02:41:20.760764 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.760737    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:20.995192 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:20.995124    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:21.059032 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:21.058995    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4fjt" event={"ID":"f3d6304c-30e6-4f4f-9327-3c4db4beac53","Type":"ContainerStarted","Data":"4ffd56045ff9135a4fed740336bb512deb6c0605fa48f246a99a6bdd3f4f3a23"}
Oct 30 02:41:21.201960 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:21.201929    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:21.553814 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:21.553783    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:21.700052 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:21.700016    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:21.906084 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:21.906009    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:22.070920 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:22.070874    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:22.241195 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:22.241169    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:22.360528 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:22.360501    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:41:22.412962 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:22.412933    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:22.584493 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:22.584465    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:23.069913 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:23.069880    5061 generic.go:356] "Generic (PLEG): container finished" podID="f3d6304c-30e6-4f4f-9327-3c4db4beac53" containerID="c945a08c65da4653a0960b41d5d70d91ad9ef599f9cf21576546a2e01cd34faa" exitCode=0
Oct 30 02:41:23.070189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:23.069969    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4fjt" event={"ID":"f3d6304c-30e6-4f4f-9327-3c4db4beac53","Type":"ContainerDied","Data":"c945a08c65da4653a0960b41d5d70d91ad9ef599f9cf21576546a2e01cd34faa"}
Oct 30 02:41:23.588108 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:23.588078    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:23.720863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:23.720814    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:24.076514 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:24.076478    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4fjt" event={"ID":"f3d6304c-30e6-4f4f-9327-3c4db4beac53","Type":"ContainerStarted","Data":"793da0855a4684a3efb11347893a17980ed901326905283c3d1341844648fa36"}
Oct 30 02:41:24.076666 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:24.076539    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4fjt" event={"ID":"f3d6304c-30e6-4f4f-9327-3c4db4beac53","Type":"ContainerStarted","Data":"be243dfae36654af7f7dff215002dd6e63d3d696076ffad8c8749f8b7556d3e5"}
Oct 30 02:41:24.136822 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:24.136760    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b4fjt" podStartSLOduration=2.762410381 podStartE2EDuration="4.136747981s" totalImagesPullingTime="1.290303256s" totalInitContainerRuntime="84.034344ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:24.126287803 +0000 UTC m=+29.428466173" watchObservedRunningTime="2026-10-30 02:41:24.136747981 +0000 UTC m=+29.438926377"
Oct 30 02:41:24.767136 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:24.767098    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:24.894682 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:24.894649    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:25.036512 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:25.036438    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:25.185579 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:25.185545    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:25.444659 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:41:25.444624    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:41:26.932920 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:26.932893    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tb8xt"
Oct 30 02:41:26.991532 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:26.991501    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:27.195737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:27.195660    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:27.395196 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:27.395159    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:28.586458 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:28.586426    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:28.717220 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:28.717186    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:28.841656 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:28.841583    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:29.872687 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:29.872651    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:30.007103 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:30.007069    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:30.165816 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:30.165743    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:31.819942 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:31.819906    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:31.958160 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:31.958093    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:32.158686 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:32.158617    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:34.411223 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:34.411190    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:34.540320 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:34.540290    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:35.234813 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:35.234734    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56b7889b4-hzhbm"]
Oct 30 02:41:35.354312 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:35.354281    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:35.472028 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:35.471995    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:35.583318 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:35.583289    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:36.164310 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:36.164282    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:36.277740 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:36.277709    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:37.379777 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.379719    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67544b9897-hb2j2" podUID="cd57c8d3-5a2c-4068-b04a-901e5795b815" containerName="registry" containerID="cri-o://84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30" gracePeriod=30
Oct 30 02:41:37.624024 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.624001    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:41:37.699660 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.699591    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-bound-sa-token\") pod \"cd57c8d3-5a2c-4068-b04a-901e5795b815\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") "
Oct 30 02:41:37.699660 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.699625    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd57c8d3-5a2c-4068-b04a-901e5795b815-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd57c8d3-5a2c-4068-b04a-901e5795b815-ca-trust-extracted\") pod \"cd57c8d3-5a2c-4068-b04a-901e5795b815\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") "
Oct 30 02:41:37.699660 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.699646    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-trusted-ca\") pod \"cd57c8d3-5a2c-4068-b04a-901e5795b815\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") "
Oct 30 02:41:37.699960 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.699665    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-installation-pull-secrets\") pod \"cd57c8d3-5a2c-4068-b04a-901e5795b815\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") "
Oct 30 02:41:37.699960 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.699746    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-kube-api-access-82nmv\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-kube-api-access-82nmv\") pod \"cd57c8d3-5a2c-4068-b04a-901e5795b815\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") "
Oct 30 02:41:37.699960 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.699803    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-image-registry-private-configuration\") pod \"cd57c8d3-5a2c-4068-b04a-901e5795b815\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") "
Oct 30 02:41:37.700122 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.699970    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\") pod \"cd57c8d3-5a2c-4068-b04a-901e5795b815\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") "
Oct 30 02:41:37.700122 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.700101    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-trusted-ca" pod "cd57c8d3-5a2c-4068-b04a-901e5795b815" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:41:37.700230 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.700106    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-certificates\") pod \"cd57c8d3-5a2c-4068-b04a-901e5795b815\" (UID: \"cd57c8d3-5a2c-4068-b04a-901e5795b815\") "
Oct 30 02:41:37.700432 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.700398    5061 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-trusted-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:41:37.700828 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.700776    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-certificates" pod "cd57c8d3-5a2c-4068-b04a-901e5795b815" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:41:37.702838 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.702811    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls" pod "cd57c8d3-5a2c-4068-b04a-901e5795b815" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:41:37.702964 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.702872    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-kube-api-access-82nmv" pod "cd57c8d3-5a2c-4068-b04a-901e5795b815" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815"). InnerVolumeSpecName "kube-api-access-82nmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:41:37.703085 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.703056    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-installation-pull-secrets" pod "cd57c8d3-5a2c-4068-b04a-901e5795b815" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:41:37.703188 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.703165    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-bound-sa-token" pod "cd57c8d3-5a2c-4068-b04a-901e5795b815" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:41:37.703226 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.703176    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-image-registry-private-configuration" pod "cd57c8d3-5a2c-4068-b04a-901e5795b815" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:41:37.709210 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.709185    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd57c8d3-5a2c-4068-b04a-901e5795b815-ca-trust-extracted" pod "cd57c8d3-5a2c-4068-b04a-901e5795b815" (UID: "cd57c8d3-5a2c-4068-b04a-901e5795b815"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:41:37.801585 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.801556    5061 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-image-registry-private-configuration\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:41:37.801585 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.801582    5061 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:41:37.801753 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.801592    5061 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd57c8d3-5a2c-4068-b04a-901e5795b815-registry-certificates\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:41:37.801753 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.801601    5061 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-bound-sa-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:41:37.801753 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.801609    5061 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd57c8d3-5a2c-4068-b04a-901e5795b815-ca-trust-extracted\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:41:37.801753 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.801618    5061 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd57c8d3-5a2c-4068-b04a-901e5795b815-installation-pull-secrets\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:41:37.801753 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:37.801626    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82nmv\" (UniqueName: \"kubernetes.io/projected/cd57c8d3-5a2c-4068-b04a-901e5795b815-kube-api-access-82nmv\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:41:38.130511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:38.130478    5061 generic.go:356] "Generic (PLEG): container finished" podID="cd57c8d3-5a2c-4068-b04a-901e5795b815" containerID="84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30" exitCode=0
Oct 30 02:41:38.130696 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:38.130533    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67544b9897-hb2j2"
Oct 30 02:41:38.130696 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:38.130536    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67544b9897-hb2j2" event={"ID":"cd57c8d3-5a2c-4068-b04a-901e5795b815","Type":"ContainerDied","Data":"84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30"}
Oct 30 02:41:38.130696 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:38.130648    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67544b9897-hb2j2" event={"ID":"cd57c8d3-5a2c-4068-b04a-901e5795b815","Type":"ContainerDied","Data":"890cddd7c633779ba5e74157fe9f6da640185b0cc116ff52e9ae965c9285bc11"}
Oct 30 02:41:38.130696 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:38.130667    5061 scope.go:122] "RemoveContainer" containerID="84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30"
Oct 30 02:41:38.139427 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:38.139407    5061 scope.go:122] "RemoveContainer" containerID="84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30"
Oct 30 02:41:38.139731 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:41:38.139711    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30\": container with ID starting with 84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30 not found: ID does not exist" containerID="84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30"
Oct 30 02:41:38.139791 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:38.139738    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30"} err="failed to get container status \"84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30\": rpc error: code = NotFound desc = could not find container \"84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30\": container with ID starting with 84ce4d28b00749c86755d0b93c5a9cd58364477182c5f32f440e8cbfd0c16a30 not found: ID does not exist"
Oct 30 02:41:38.190821 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:38.190796    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67544b9897-hb2j2"]
Oct 30 02:41:38.201747 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:38.201727    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67544b9897-hb2j2"]
Oct 30 02:41:38.844604 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:38.844568    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd57c8d3-5a2c-4068-b04a-901e5795b815" path="/var/lib/kubelet/pods/cd57c8d3-5a2c-4068-b04a-901e5795b815/volumes"
Oct 30 02:41:44.118271 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:44.118224    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:44.260003 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:44.259966    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:44.839533 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:44.839479    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:44.958685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:44.958648    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:55.444528 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:41:55.444481    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:41:59.216975 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.216891    5061 generic.go:356] "Generic (PLEG): container finished" podID="76243024-183c-42e6-89fd-ccd82d3571c4" containerID="e758dd753f771f94e9471c3230c808507ea3d3b1d7cca0eabfb5fab77efcec2a" exitCode=0
Oct 30 02:41:59.217326 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.216967    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zh9rv" event={"ID":"76243024-183c-42e6-89fd-ccd82d3571c4","Type":"ContainerDied","Data":"e758dd753f771f94e9471c3230c808507ea3d3b1d7cca0eabfb5fab77efcec2a"}
Oct 30 02:41:59.217326 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.217272    5061 scope.go:122] "RemoveContainer" containerID="e758dd753f771f94e9471c3230c808507ea3d3b1d7cca0eabfb5fab77efcec2a"
Oct 30 02:41:59.692804 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.692765    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-q8xpw-hdnw7"]
Oct 30 02:41:59.693033 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.692997    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-q8xpw-hdnw7" podUID="d786d081-b91d-475a-937b-021d17c210b1" containerName="insights-gathering" containerID="cri-o://9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659" gracePeriod=30
Oct 30 02:41:59.877697 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.877669    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bsd2d-997t6"]
Oct 30 02:41:59.878093 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.878070    5061 cpu_manager.go:414] "containerMap: removing container" podUID="cd57c8d3-5a2c-4068-b04a-901e5795b815" containerName="registry"
Oct 30 02:41:59.878093 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.878092    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd57c8d3-5a2c-4068-b04a-901e5795b815" containerName="registry"
Oct 30 02:41:59.878276 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.878194    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd57c8d3-5a2c-4068-b04a-901e5795b815" containerName="registry"
Oct 30 02:41:59.880359 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.880339    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:41:59.899946 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.899880    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bsd2d-997t6"]
Oct 30 02:41:59.941274 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.941252    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:41:59.970374 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.970316    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-service-ca-bundle\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:41:59.970374 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.970354    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mc5n\" (UniqueName: \"kubernetes.io/projected/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-kube-api-access-4mc5n\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:41:59.970505 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.970375    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-archives-path\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:41:59.970505 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:41:59.970433    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-trusted-ca-bundle\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:42:00.071627 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.071596    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-trusted-ca-bundle\") pod \"d786d081-b91d-475a-937b-021d17c210b1\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") "
Oct 30 02:42:00.071800 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.071633    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d786d081-b91d-475a-937b-021d17c210b1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d786d081-b91d-475a-937b-021d17c210b1-archives-path\") pod \"d786d081-b91d-475a-937b-021d17c210b1\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") "
Oct 30 02:42:00.071800 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.071657    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-service-ca-bundle\") pod \"d786d081-b91d-475a-937b-021d17c210b1\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") "
Oct 30 02:42:00.071800 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.071687    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d786d081-b91d-475a-937b-021d17c210b1-kube-api-access-b5wg8\" (UniqueName: \"kubernetes.io/projected/d786d081-b91d-475a-937b-021d17c210b1-kube-api-access-b5wg8\") pod \"d786d081-b91d-475a-937b-021d17c210b1\" (UID: \"d786d081-b91d-475a-937b-021d17c210b1\") "
Oct 30 02:42:00.071800 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.071758    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mc5n\" (UniqueName: \"kubernetes.io/projected/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-kube-api-access-4mc5n\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:42:00.071800 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.071782    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-archives-path\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:42:00.072099 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.071820    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-trusted-ca-bundle\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:42:00.072099 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.072020    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d786d081-b91d-475a-937b-021d17c210b1-archives-path" pod "d786d081-b91d-475a-937b-021d17c210b1" (UID: "d786d081-b91d-475a-937b-021d17c210b1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:00.072099 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.072078    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-service-ca-bundle\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:42:00.072227 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.072155    5061 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d786d081-b91d-475a-937b-021d17c210b1-archives-path\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.072227 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.072078    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-service-ca-bundle" pod "d786d081-b91d-475a-937b-021d17c210b1" (UID: "d786d081-b91d-475a-937b-021d17c210b1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:00.072227 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.072110    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-trusted-ca-bundle" pod "d786d081-b91d-475a-937b-021d17c210b1" (UID: "d786d081-b91d-475a-937b-021d17c210b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:00.072345 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.072324    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-archives-path\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:42:00.072621 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.072603    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-service-ca-bundle\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:42:00.072836 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.072810    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-trusted-ca-bundle\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:42:00.073980 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.073961    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d786d081-b91d-475a-937b-021d17c210b1-kube-api-access-b5wg8" pod "d786d081-b91d-475a-937b-021d17c210b1" (UID: "d786d081-b91d-475a-937b-021d17c210b1"). InnerVolumeSpecName "kube-api-access-b5wg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:00.090124 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.090097    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mc5n\" (UniqueName: \"kubernetes.io/projected/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-kube-api-access-4mc5n\") pod \"periodic-gathering-bsd2d-997t6\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") " pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:42:00.173217 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.173193    5061 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-trusted-ca-bundle\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.173217 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.173215    5061 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d786d081-b91d-475a-937b-021d17c210b1-service-ca-bundle\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.173342 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.173225    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5wg8\" (UniqueName: \"kubernetes.io/projected/d786d081-b91d-475a-937b-021d17c210b1-kube-api-access-b5wg8\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.191176 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.191147    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:42:00.223141 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.223063    5061 generic.go:356] "Generic (PLEG): container finished" podID="d786d081-b91d-475a-937b-021d17c210b1" containerID="9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659" exitCode=2
Oct 30 02:42:00.223141 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.223111    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q8xpw-hdnw7"
Oct 30 02:42:00.223556 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.223153    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8xpw-hdnw7" event={"ID":"d786d081-b91d-475a-937b-021d17c210b1","Type":"ContainerDied","Data":"9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659"}
Oct 30 02:42:00.223556 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.223212    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q8xpw-hdnw7" event={"ID":"d786d081-b91d-475a-937b-021d17c210b1","Type":"ContainerDied","Data":"c161763cc1a81d0ec45280c251660d734c6ee1894a175d8969c2ab828452effa"}
Oct 30 02:42:00.223556 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.223244    5061 scope.go:122] "RemoveContainer" containerID="9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659"
Oct 30 02:42:00.225300 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.225276    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zh9rv" event={"ID":"76243024-183c-42e6-89fd-ccd82d3571c4","Type":"ContainerStarted","Data":"37c3ecf3ecf812baa0d58be303b5da255fdcdb0330a1d5368ec4b3c797a8c573"}
Oct 30 02:42:00.232428 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.232410    5061 scope.go:122] "RemoveContainer" containerID="9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659"
Oct 30 02:42:00.232761 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:42:00.232739    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659\": container with ID starting with 9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659 not found: ID does not exist" containerID="9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659"
Oct 30 02:42:00.232873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.232768    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659"} err="failed to get container status \"9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659\": rpc error: code = NotFound desc = could not find container \"9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659\": container with ID starting with 9ef33069a07580f443f7cdc9f633f5d0e0d71eb891727b797348e2e48a49f659 not found: ID does not exist"
Oct 30 02:42:00.259150 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.259105    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm" podUID="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" containerName="registry" containerID="cri-o://7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8" gracePeriod=30
Oct 30 02:42:00.269032 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.269007    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-q8xpw-hdnw7"]
Oct 30 02:42:00.271074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.271046    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-q8xpw-hdnw7"]
Oct 30 02:42:00.309721 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.309692    5061 patch_prober.go:29] interesting pod/image-registry-56b7889b4-hzhbm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.22:5000/healthz\": dial tcp 10.132.0.22:5000: connect: connection refused" start-of-body=
Oct 30 02:42:00.309867 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.309739    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm" podUID="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.22:5000/healthz\": dial tcp 10.132.0.22:5000: connect: connection refused"
Oct 30 02:42:00.475320 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.475266    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:42:00.492261 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.492238    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bsd2d-997t6"]
Oct 30 02:42:00.496259 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:42:00.496239    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode546f73e_8efa_47e9_9bfb_a2c982dff9c4.slice/crio-0d5b741849f2c6f717500847ff2db7ad5c08b98e15f34a89ba900564238d77b3 WatchSource:0}: Error finding container 0d5b741849f2c6f717500847ff2db7ad5c08b98e15f34a89ba900564238d77b3: Status 404 returned error can't find the container with id 0d5b741849f2c6f717500847ff2db7ad5c08b98e15f34a89ba900564238d77b3
Oct 30 02:42:00.576358 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.576333    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-image-registry-private-configuration\") pod \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") "
Oct 30 02:42:00.576508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.576374    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-ca-trust-extracted\") pod \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") "
Oct 30 02:42:00.576508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.576395    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-installation-pull-secrets\") pod \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") "
Oct 30 02:42:00.576508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.576427    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\") pod \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") "
Oct 30 02:42:00.576508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.576465    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-trusted-ca\") pod \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") "
Oct 30 02:42:00.576508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.576505    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-bound-sa-token\") pod \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") "
Oct 30 02:42:00.576782 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.576549    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-certificates\") pod \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") "
Oct 30 02:42:00.576782 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.576589    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-kube-api-access-dfg6j\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-kube-api-access-dfg6j\") pod \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\" (UID: \"4f8a405e-f80a-4aa9-ab4c-2217e9250b70\") "
Oct 30 02:42:00.577634 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.577125    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-certificates" pod "4f8a405e-f80a-4aa9-ab4c-2217e9250b70" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:00.577634 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.577348    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-trusted-ca" pod "4f8a405e-f80a-4aa9-ab4c-2217e9250b70" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:00.579131 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.579104    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls" pod "4f8a405e-f80a-4aa9-ab4c-2217e9250b70" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:00.579257 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.579174    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-image-registry-private-configuration" pod "4f8a405e-f80a-4aa9-ab4c-2217e9250b70" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:00.579326 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.579278    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-kube-api-access-dfg6j" pod "4f8a405e-f80a-4aa9-ab4c-2217e9250b70" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70"). InnerVolumeSpecName "kube-api-access-dfg6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:00.579598 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.579572    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-bound-sa-token" pod "4f8a405e-f80a-4aa9-ab4c-2217e9250b70" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:00.580181 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.580156    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-installation-pull-secrets" pod "4f8a405e-f80a-4aa9-ab4c-2217e9250b70" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:00.584130 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.584108    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-ca-trust-extracted" pod "4f8a405e-f80a-4aa9-ab4c-2217e9250b70" (UID: "4f8a405e-f80a-4aa9-ab4c-2217e9250b70"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:00.677945 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.677912    5061 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-image-registry-private-configuration\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.677945 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.677940    5061 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-ca-trust-extracted\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.677945 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.677950    5061 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-installation-pull-secrets\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.678158 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.677960    5061 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.678158 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.677968    5061 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-trusted-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.678158 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.677977    5061 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-bound-sa-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.678158 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.677986    5061 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-registry-certificates\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.678158 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.677996    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfg6j\" (UniqueName: \"kubernetes.io/projected/4f8a405e-f80a-4aa9-ab4c-2217e9250b70-kube-api-access-dfg6j\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:42:00.846910 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:00.846879    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d786d081-b91d-475a-937b-021d17c210b1" path="/var/lib/kubelet/pods/d786d081-b91d-475a-937b-021d17c210b1/volumes"
Oct 30 02:42:01.038671 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.038639    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b959b9795-jw4rz_6c28bc38-2c97-412f-89b0-f7a552b78373/router/0.log"
Oct 30 02:42:01.155551 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.155484    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9r4jn_e0334aa2-bc83-440a-8781-60817828e1ab/serve-healthcheck-canary/0.log"
Oct 30 02:42:01.230290 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.230262    5061 generic.go:356] "Generic (PLEG): container finished" podID="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" containerID="7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8" exitCode=0
Oct 30 02:42:01.230689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.230314    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm"
Oct 30 02:42:01.230689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.230346    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm" event={"ID":"4f8a405e-f80a-4aa9-ab4c-2217e9250b70","Type":"ContainerDied","Data":"7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8"}
Oct 30 02:42:01.230689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.230456    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b7889b4-hzhbm" event={"ID":"4f8a405e-f80a-4aa9-ab4c-2217e9250b70","Type":"ContainerDied","Data":"a41a695233fece2f22d5fd0da949b94c813bb3e8613e1ef49a0fd744c4dac412"}
Oct 30 02:42:01.230689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.230488    5061 scope.go:122] "RemoveContainer" containerID="7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8"
Oct 30 02:42:01.231991 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.231961    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bsd2d-997t6" event={"ID":"e546f73e-8efa-47e9-9bfb-a2c982dff9c4","Type":"ContainerStarted","Data":"5cc7d17d34c0784333f42e467f79f3b7188aaca359f0974e1a4755fb3a1c692b"}
Oct 30 02:42:01.232088 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.232015    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bsd2d-997t6" event={"ID":"e546f73e-8efa-47e9-9bfb-a2c982dff9c4","Type":"ContainerStarted","Data":"0d5b741849f2c6f717500847ff2db7ad5c08b98e15f34a89ba900564238d77b3"}
Oct 30 02:42:01.238810 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.238790    5061 scope.go:122] "RemoveContainer" containerID="7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8"
Oct 30 02:42:01.239110 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:42:01.239088    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8\": container with ID starting with 7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8 not found: ID does not exist" containerID="7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8"
Oct 30 02:42:01.239191 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.239115    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8"} err="failed to get container status \"7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8\": rpc error: code = NotFound desc = could not find container \"7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8\": container with ID starting with 7a7b56f0bae80cd99eef04b3e6d163824f197dfb946c202166214e1ef363a3c8 not found: ID does not exist"
Oct 30 02:42:01.286876 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.286857    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56b7889b4-hzhbm"]
Oct 30 02:42:01.297623 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.297606    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56b7889b4-hzhbm"]
Oct 30 02:42:01.330932 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:01.330885    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bsd2d-997t6" podStartSLOduration=2.330870924 podStartE2EDuration="2.330870924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:01.329725263 +0000 UTC m=+66.631903640" watchObservedRunningTime="2026-10-30 02:42:01.330870924 +0000 UTC m=+66.633049290"
Oct 30 02:42:02.844688 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:02.844657    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" path="/var/lib/kubelet/pods/4f8a405e-f80a-4aa9-ab4c-2217e9250b70/volumes"
Oct 30 02:42:05.879305 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:05.879269    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:05.987096 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:05.987069    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:07.724798 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:07.724764    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:07.864339 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:07.864305    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:25.446909 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:42:25.446866    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:42:37.035210 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:37.035173    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:37.162982 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:37.162950    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:44.270060 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:44.270025    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:47.357023 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:47.356990    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:47.481790 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:47.481762    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:47.638042 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:47.637975    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:47.751808 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:47.751783    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:49.680841 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:49.680806    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:49.806727 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:49.806696    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:55.444339 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:42:55.444292    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:42:59.835911 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:59.835876    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:59.953999 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:42:59.953962    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:25.444376 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:43:25.444339    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:43:26.567449 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:26.567415    5061 generic.go:356] "Generic (PLEG): container finished" podID="e546f73e-8efa-47e9-9bfb-a2c982dff9c4" containerID="5cc7d17d34c0784333f42e467f79f3b7188aaca359f0974e1a4755fb3a1c692b" exitCode=0
Oct 30 02:43:26.567810 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:26.567459    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bsd2d-997t6" event={"ID":"e546f73e-8efa-47e9-9bfb-a2c982dff9c4","Type":"ContainerDied","Data":"5cc7d17d34c0784333f42e467f79f3b7188aaca359f0974e1a4755fb3a1c692b"}
Oct 30 02:43:27.701021 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.700997    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:43:27.839343 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.839260    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-kube-api-access-4mc5n\" (UniqueName: \"kubernetes.io/projected/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-kube-api-access-4mc5n\") pod \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") "
Oct 30 02:43:27.839343 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.839309    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-service-ca-bundle\") pod \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") "
Oct 30 02:43:27.839343 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.839334    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-trusted-ca-bundle\") pod \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") "
Oct 30 02:43:27.839609 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.839353    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-archives-path\") pod \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\" (UID: \"e546f73e-8efa-47e9-9bfb-a2c982dff9c4\") "
Oct 30 02:43:27.839741 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.839711    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-archives-path" pod "e546f73e-8efa-47e9-9bfb-a2c982dff9c4" (UID: "e546f73e-8efa-47e9-9bfb-a2c982dff9c4"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:43:27.839741 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.839716    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-service-ca-bundle" pod "e546f73e-8efa-47e9-9bfb-a2c982dff9c4" (UID: "e546f73e-8efa-47e9-9bfb-a2c982dff9c4"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:27.839958 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.839750    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-trusted-ca-bundle" pod "e546f73e-8efa-47e9-9bfb-a2c982dff9c4" (UID: "e546f73e-8efa-47e9-9bfb-a2c982dff9c4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:27.841397 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.841376    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-kube-api-access-4mc5n" pod "e546f73e-8efa-47e9-9bfb-a2c982dff9c4" (UID: "e546f73e-8efa-47e9-9bfb-a2c982dff9c4"). InnerVolumeSpecName "kube-api-access-4mc5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:43:27.939924 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.939897    5061 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-service-ca-bundle\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:43:27.939924 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.939923    5061 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-trusted-ca-bundle\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:43:27.940060 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.939938    5061 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-archives-path\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:43:27.940060 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:27.939953    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mc5n\" (UniqueName: \"kubernetes.io/projected/e546f73e-8efa-47e9-9bfb-a2c982dff9c4-kube-api-access-4mc5n\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:43:28.575714 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:28.575681    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bsd2d-997t6" event={"ID":"e546f73e-8efa-47e9-9bfb-a2c982dff9c4","Type":"ContainerDied","Data":"0d5b741849f2c6f717500847ff2db7ad5c08b98e15f34a89ba900564238d77b3"}
Oct 30 02:43:28.575896 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:28.575725    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bsd2d-997t6"
Oct 30 02:43:28.575896 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:28.575734    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d5b741849f2c6f717500847ff2db7ad5c08b98e15f34a89ba900564238d77b3"
Oct 30 02:43:33.138870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:33.138817    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:33.263999 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:33.263968    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:44.382891 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:44.382836    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:55.444542 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:43:55.444506    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:43:59.933964 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:43:59.933936    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:00.047143 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:44:00.047115    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:25.444742 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:44:25.444692    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:44:44.492617 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:44:44.492535    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:53.763592 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:44:53.763533    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:53.908504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:44:53.908468    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:55.444271 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:44:55.444226    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:45:15.522549 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:15.522502    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:17.621256 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:17.621227    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:18.714961 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:18.714928    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:18.833132 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:18.833103    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:21.515968 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:21.515935    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:21.688210 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:21.688179    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:23.818175 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:23.818140    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:24.054887 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:24.054840    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:25.444724 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:45:25.444681    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:45:25.731097 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:25.731020    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:25.856409 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:25.856376    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:29.721301 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:29.721269    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:29.833702 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:29.833667    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:31.043331 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.043297    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:31.823661 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.823630    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-q5pjs"]
Oct 30 02:45:31.823940 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.823927    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e546f73e-8efa-47e9-9bfb-a2c982dff9c4" containerName="insights-gathering"
Oct 30 02:45:31.823990 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.823942    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e546f73e-8efa-47e9-9bfb-a2c982dff9c4" containerName="insights-gathering"
Oct 30 02:45:31.823990 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.823952    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d786d081-b91d-475a-937b-021d17c210b1" containerName="insights-gathering"
Oct 30 02:45:31.823990 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.823957    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d786d081-b91d-475a-937b-021d17c210b1" containerName="insights-gathering"
Oct 30 02:45:31.823990 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.823988    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" containerName="registry"
Oct 30 02:45:31.824116 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.823993    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" containerName="registry"
Oct 30 02:45:31.824116 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.824040    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8a405e-f80a-4aa9-ab4c-2217e9250b70" containerName="registry"
Oct 30 02:45:31.824116 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.824047    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e546f73e-8efa-47e9-9bfb-a2c982dff9c4" containerName="insights-gathering"
Oct 30 02:45:31.824116 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.824052    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d786d081-b91d-475a-937b-021d17c210b1" containerName="insights-gathering"
Oct 30 02:45:31.826925 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.826905    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-q5pjs"
Oct 30 02:45:31.857741 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.857712    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-fwjjd\""
Oct 30 02:45:31.858019 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.858002    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 02:45:31.858107 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.858039    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 02:45:31.859088 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.859064    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-q5pjs"]
Oct 30 02:45:31.926646 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.926619    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgxdg\" (UniqueName: \"kubernetes.io/projected/f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72-kube-api-access-lgxdg\") pod \"cert-manager-597b95cb84-q5pjs\" (UID: \"f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72\") " pod="cert-manager/cert-manager-597b95cb84-q5pjs"
Oct 30 02:45:31.926749 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:31.926673    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72-bound-sa-token\") pod \"cert-manager-597b95cb84-q5pjs\" (UID: \"f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72\") " pod="cert-manager/cert-manager-597b95cb84-q5pjs"
Oct 30 02:45:32.027703 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:32.027675    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgxdg\" (UniqueName: \"kubernetes.io/projected/f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72-kube-api-access-lgxdg\") pod \"cert-manager-597b95cb84-q5pjs\" (UID: \"f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72\") " pod="cert-manager/cert-manager-597b95cb84-q5pjs"
Oct 30 02:45:32.027799 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:32.027719    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72-bound-sa-token\") pod \"cert-manager-597b95cb84-q5pjs\" (UID: \"f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72\") " pod="cert-manager/cert-manager-597b95cb84-q5pjs"
Oct 30 02:45:32.045538 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:32.045518    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgxdg\" (UniqueName: \"kubernetes.io/projected/f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72-kube-api-access-lgxdg\") pod \"cert-manager-597b95cb84-q5pjs\" (UID: \"f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72\") " pod="cert-manager/cert-manager-597b95cb84-q5pjs"
Oct 30 02:45:32.045944 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:32.045581    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72-bound-sa-token\") pod \"cert-manager-597b95cb84-q5pjs\" (UID: \"f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72\") " pod="cert-manager/cert-manager-597b95cb84-q5pjs"
Oct 30 02:45:32.153836 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:32.153771    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-q5pjs"
Oct 30 02:45:32.442122 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:32.442048    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-q5pjs"]
Oct 30 02:45:32.447219 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:45:32.447198    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4acef26_c2f2_42dd_a6ae_e2cf4a9b0e72.slice/crio-da699e0ca76161bb096b92dd4bd61499b7eeaf779af795c4ddcb8877867a32e5 WatchSource:0}: Error finding container da699e0ca76161bb096b92dd4bd61499b7eeaf779af795c4ddcb8877867a32e5: Status 404 returned error can't find the container with id da699e0ca76161bb096b92dd4bd61499b7eeaf779af795c4ddcb8877867a32e5
Oct 30 02:45:32.935804 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:32.935772    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:33.050871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:33.050827    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-q5pjs" event={"ID":"f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72","Type":"ContainerStarted","Data":"da699e0ca76161bb096b92dd4bd61499b7eeaf779af795c4ddcb8877867a32e5"}
Oct 30 02:45:33.052737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:33.052713    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:36.066196 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:36.066163    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-q5pjs" event={"ID":"f4acef26-c2f2-42dd-a6ae-e2cf4a9b0e72","Type":"ContainerStarted","Data":"f58c0fee39765400aa301bcd1574f2dd59070470a3c11a7a6c9d7a684f1eee23"}
Oct 30 02:45:36.102010 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:36.101962    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-q5pjs" podStartSLOduration=1.695361558 podStartE2EDuration="5.101949676s" totalImagesPullingTime="3.406588118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:36.100213337 +0000 UTC m=+281.402391701" watchObservedRunningTime="2026-10-30 02:45:36.101949676 +0000 UTC m=+281.404128040"
Oct 30 02:45:39.282623 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:39.282587    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:39.398337 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:39.398301    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:41.773333 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:41.773300    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:44.594005 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:44.593973    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:51.728738 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:51.728706    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:51.844620 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:51.844589    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:52.292345 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.292316    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:52.445041 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.445002    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"]
Oct 30 02:45:52.451791 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.451766    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.469940 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.469915    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 02:45:52.470105 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.470088    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 02:45:52.477126 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.477108    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 02:45:52.477232 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.477214    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 02:45:52.477571 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.477554    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-knctd\""
Oct 30 02:45:52.477571 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.477565    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 02:45:52.482138 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.482117    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28c8ee4f-5f39-4539-abc8-efc5679db424-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.482226 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.482159    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcxtm\" (UniqueName: \"kubernetes.io/projected/28c8ee4f-5f39-4539-abc8-efc5679db424-kube-api-access-tcxtm\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.482226 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.482188    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28c8ee4f-5f39-4539-abc8-efc5679db424-manifests\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.482317 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.482298    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28c8ee4f-5f39-4539-abc8-efc5679db424-metrics-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.482351 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.482327    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28c8ee4f-5f39-4539-abc8-efc5679db424-webhook-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.482383 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.482357    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28c8ee4f-5f39-4539-abc8-efc5679db424-tmp\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.493764 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.493744    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"]
Oct 30 02:45:52.582904 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.582872    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28c8ee4f-5f39-4539-abc8-efc5679db424-manifests\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.583078 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.582922    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28c8ee4f-5f39-4539-abc8-efc5679db424-metrics-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.583078 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.582971    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28c8ee4f-5f39-4539-abc8-efc5679db424-webhook-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.583078 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.583020    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28c8ee4f-5f39-4539-abc8-efc5679db424-tmp\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.583078 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.583067    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28c8ee4f-5f39-4539-abc8-efc5679db424-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.583293 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.583096    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcxtm\" (UniqueName: \"kubernetes.io/projected/28c8ee4f-5f39-4539-abc8-efc5679db424-kube-api-access-tcxtm\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.583677 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.583620    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28c8ee4f-5f39-4539-abc8-efc5679db424-manifests\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.585386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.585359    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28c8ee4f-5f39-4539-abc8-efc5679db424-tmp\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.585508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.585458    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28c8ee4f-5f39-4539-abc8-efc5679db424-metrics-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.585508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.585471    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28c8ee4f-5f39-4539-abc8-efc5679db424-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.585508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.585475    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28c8ee4f-5f39-4539-abc8-efc5679db424-webhook-cert\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.600279 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.600259    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcxtm\" (UniqueName: \"kubernetes.io/projected/28c8ee4f-5f39-4539-abc8-efc5679db424-kube-api-access-tcxtm\") pod \"opendatahub-operator-controller-manager-75bcb9cf-5hfrf\" (UID: \"28c8ee4f-5f39-4539-abc8-efc5679db424\") " pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.605144 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.605111    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:52.729359 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.729335    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:52.767507 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.767481    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:52.852655 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:52.852591    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:53.200870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:53.200779    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"]
Oct 30 02:45:53.215050 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:45:53.215025    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c8ee4f_5f39_4539_abc8_efc5679db424.slice/crio-bb4e3478fb0c232b7d1fa0b63da80710dd427d532a777313168bc01950a1185e WatchSource:0}: Error finding container bb4e3478fb0c232b7d1fa0b63da80710dd427d532a777313168bc01950a1185e: Status 404 returned error can't find the container with id bb4e3478fb0c232b7d1fa0b63da80710dd427d532a777313168bc01950a1185e
Oct 30 02:45:53.280272 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:53.280243    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:53.397436 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:53.397401    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:54.136719 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:54.136678    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf" event={"ID":"28c8ee4f-5f39-4539-abc8-efc5679db424","Type":"ContainerStarted","Data":"bb4e3478fb0c232b7d1fa0b63da80710dd427d532a777313168bc01950a1185e"}
Oct 30 02:45:54.865752 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:54.865724    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:45:54.865965 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:54.865954    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:45:54.868475 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:54.868457    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:45:54.868675 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:54.868663    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:45:55.447049 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:45:55.447014    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:45:56.146527 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:56.146501    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf" event={"ID":"28c8ee4f-5f39-4539-abc8-efc5679db424","Type":"ContainerStarted","Data":"d6b3781897d7c2e3edb534c7001d71ed91e0874a1c75d5873b98b8581e35e1da"}
Oct 30 02:45:57.152008 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:57.151970    5061 generic.go:356] "Generic (PLEG): container finished" podID="28c8ee4f-5f39-4539-abc8-efc5679db424" containerID="d6b3781897d7c2e3edb534c7001d71ed91e0874a1c75d5873b98b8581e35e1da" exitCode=0
Oct 30 02:45:57.152414 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:57.152054    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf" event={"ID":"28c8ee4f-5f39-4539-abc8-efc5679db424","Type":"ContainerDied","Data":"d6b3781897d7c2e3edb534c7001d71ed91e0874a1c75d5873b98b8581e35e1da"}
Oct 30 02:45:57.152536 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:57.152521    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:45:58.158510 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:58.158469    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf" event={"ID":"28c8ee4f-5f39-4539-abc8-efc5679db424","Type":"ContainerStarted","Data":"cfec7c7f2db78ac798b233897ccc052b5c9faa619871a64c351f7187c67814be"}
Oct 30 02:45:58.158965 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:58.158609    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:45:58.227488 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:45:58.227425    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf" podStartSLOduration=2.523772858 podStartE2EDuration="6.227407046s" totalImagesPullingTime="3.218600186s" totalInitContainerRuntime="485.034002ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:58.21630944 +0000 UTC m=+303.518487805" watchObservedRunningTime="2026-10-30 02:45:58.227407046 +0000 UTC m=+303.529585412"
Oct 30 02:46:06.266094 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:06.266011    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:06.416511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:06.416481    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:06.531834 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:06.531732    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:09.165839 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:09.165802    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-75bcb9cf-5hfrf"
Oct 30 02:46:11.840842 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:11.840806    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:11.959053 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:11.959022    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:12.261755 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.261686    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64f884f994-jv42m"]
Oct 30 02:46:12.265165 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.265145    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m"
Oct 30 02:46:12.279040 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.279014    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 02:46:12.282083 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.282064    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fv4st\""
Oct 30 02:46:12.294285 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.294263    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64f884f994-jv42m"]
Oct 30 02:46:12.299098 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.299079    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:12.323739 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.323723    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 02:46:12.330307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.330286    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eda807c-afcf-48fb-adee-e8881850f7a8-tls-certs\") pod \"kube-auth-proxy-64f884f994-jv42m\" (UID: \"9eda807c-afcf-48fb-adee-e8881850f7a8\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m"
Oct 30 02:46:12.330419 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.330326    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eda807c-afcf-48fb-adee-e8881850f7a8-tmp\") pod \"kube-auth-proxy-64f884f994-jv42m\" (UID: \"9eda807c-afcf-48fb-adee-e8881850f7a8\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m"
Oct 30 02:46:12.330419 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.330371    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvqh\" (UniqueName: \"kubernetes.io/projected/9eda807c-afcf-48fb-adee-e8881850f7a8-kube-api-access-sbvqh\") pod \"kube-auth-proxy-64f884f994-jv42m\" (UID: \"9eda807c-afcf-48fb-adee-e8881850f7a8\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m"
Oct 30 02:46:12.428010 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.427967    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:12.431079 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.431057    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eda807c-afcf-48fb-adee-e8881850f7a8-tls-certs\") pod \"kube-auth-proxy-64f884f994-jv42m\" (UID: \"9eda807c-afcf-48fb-adee-e8881850f7a8\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m"
Oct 30 02:46:12.431190 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.431111    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eda807c-afcf-48fb-adee-e8881850f7a8-tmp\") pod \"kube-auth-proxy-64f884f994-jv42m\" (UID: \"9eda807c-afcf-48fb-adee-e8881850f7a8\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m"
Oct 30 02:46:12.431190 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.431185    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvqh\" (UniqueName: \"kubernetes.io/projected/9eda807c-afcf-48fb-adee-e8881850f7a8-kube-api-access-sbvqh\") pod \"kube-auth-proxy-64f884f994-jv42m\" (UID: \"9eda807c-afcf-48fb-adee-e8881850f7a8\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m"
Oct 30 02:46:12.433389 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.433370    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eda807c-afcf-48fb-adee-e8881850f7a8-tmp\") pod \"kube-auth-proxy-64f884f994-jv42m\" (UID: \"9eda807c-afcf-48fb-adee-e8881850f7a8\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m"
Oct 30 02:46:12.433625 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.433602    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eda807c-afcf-48fb-adee-e8881850f7a8-tls-certs\") pod \"kube-auth-proxy-64f884f994-jv42m\" (UID: \"9eda807c-afcf-48fb-adee-e8881850f7a8\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m"
Oct 30 02:46:12.449327 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.449296    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvqh\" (UniqueName: \"kubernetes.io/projected/9eda807c-afcf-48fb-adee-e8881850f7a8-kube-api-access-sbvqh\") pod \"kube-auth-proxy-64f884f994-jv42m\" (UID: \"9eda807c-afcf-48fb-adee-e8881850f7a8\") " pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m"
Oct 30 02:46:12.578066 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.578030    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m"
Oct 30 02:46:12.880863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:12.880773    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64f884f994-jv42m"]
Oct 30 02:46:12.884877 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:46:12.884836    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eda807c_afcf_48fb_adee_e8881850f7a8.slice/crio-a5cc35b07422d31a1caea4bd7238405809a22a2cb743888da3f7339bb46bdc22 WatchSource:0}: Error finding container a5cc35b07422d31a1caea4bd7238405809a22a2cb743888da3f7339bb46bdc22: Status 404 returned error can't find the container with id a5cc35b07422d31a1caea4bd7238405809a22a2cb743888da3f7339bb46bdc22
Oct 30 02:46:13.222631 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:13.222557    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m" event={"ID":"9eda807c-afcf-48fb-adee-e8881850f7a8","Type":"ContainerStarted","Data":"a5cc35b07422d31a1caea4bd7238405809a22a2cb743888da3f7339bb46bdc22"}
Oct 30 02:46:13.669937 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:13.669832    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:13.809509 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:13.809473    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:14.605891 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.605839    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:14.676321 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.676283    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"]
Oct 30 02:46:14.685554 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.685527    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.697877 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.697797    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"]
Oct 30 02:46:14.755050 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.755016    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd86h\" (UniqueName: \"kubernetes.io/projected/2b3ac8ea-507d-4195-988e-6128a45cc315-kube-api-access-zd86h\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.755230 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.755084    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b3ac8ea-507d-4195-988e-6128a45cc315-cert\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.755230 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.755180    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2b3ac8ea-507d-4195-988e-6128a45cc315-manager-config\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.755342 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.755247    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b3ac8ea-507d-4195-988e-6128a45cc315-metrics-cert\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.790502 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.790472    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 02:46:14.790502 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.790481    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:46:14.790772 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.790619    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kqx47\""
Oct 30 02:46:14.790772 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.790649    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 02:46:14.790772 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.790668    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 02:46:14.790772 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.790696    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 02:46:14.856776 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.856680    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b3ac8ea-507d-4195-988e-6128a45cc315-metrics-cert\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.856776 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.856762    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd86h\" (UniqueName: \"kubernetes.io/projected/2b3ac8ea-507d-4195-988e-6128a45cc315-kube-api-access-zd86h\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.856965 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.856820    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b3ac8ea-507d-4195-988e-6128a45cc315-cert\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.856965 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.856878    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2b3ac8ea-507d-4195-988e-6128a45cc315-manager-config\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.857524 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.857495    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2b3ac8ea-507d-4195-988e-6128a45cc315-manager-config\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.859610 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.859590    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b3ac8ea-507d-4195-988e-6128a45cc315-metrics-cert\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.859707 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.859679    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b3ac8ea-507d-4195-988e-6128a45cc315-cert\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:14.873807 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:14.873787    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd86h\" (UniqueName: \"kubernetes.io/projected/2b3ac8ea-507d-4195-988e-6128a45cc315-kube-api-access-zd86h\") pod \"lws-controller-manager-5548b4dc5c-pmdmw\" (UID: \"2b3ac8ea-507d-4195-988e-6128a45cc315\") " pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:15.000741 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:15.000699    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:15.141291 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:15.141199    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:15.301901 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:15.301869    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"]
Oct 30 02:46:15.307033 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:46:15.307004    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3ac8ea_507d_4195_988e_6128a45cc315.slice/crio-a7af4a2c62537e1dd9884dd24a3e8886f99f830937535c193d3a034b6a874c39 WatchSource:0}: Error finding container a7af4a2c62537e1dd9884dd24a3e8886f99f830937535c193d3a034b6a874c39: Status 404 returned error can't find the container with id a7af4a2c62537e1dd9884dd24a3e8886f99f830937535c193d3a034b6a874c39
Oct 30 02:46:15.443154 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:15.443123    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:15.773048 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:15.772964    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:16.239624 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:16.239585    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m" event={"ID":"9eda807c-afcf-48fb-adee-e8881850f7a8","Type":"ContainerStarted","Data":"c199b34964ce0ac7fe55c00f8f01854e296383c51dddc348cd1404735cbfa7e0"}
Oct 30 02:46:16.240787 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:16.240764    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw" event={"ID":"2b3ac8ea-507d-4195-988e-6128a45cc315","Type":"ContainerStarted","Data":"a7af4a2c62537e1dd9884dd24a3e8886f99f830937535c193d3a034b6a874c39"}
Oct 30 02:46:16.275004 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:16.274961    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64f884f994-jv42m" podStartSLOduration=1.7741705859999999 podStartE2EDuration="4.274947805s" totalImagesPullingTime="2.500777219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:46:16.274425384 +0000 UTC m=+321.576603749" watchObservedRunningTime="2026-10-30 02:46:16.274947805 +0000 UTC m=+321.577126223"
Oct 30 02:46:16.307595 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:16.307530    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:16.410157 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:16.410125    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:17.295104 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:17.295068    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:17.417998 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:17.417962    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:17.548431 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:17.548352    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:18.035323 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:18.035249    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:18.152195 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:18.152149    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:18.330248 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:18.330214    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:19.255284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:19.255245    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw" event={"ID":"2b3ac8ea-507d-4195-988e-6128a45cc315","Type":"ContainerStarted","Data":"022d0e962ec3f09eda699ff60f01db0beec3c9a60d701acf0e215bf075037af7"}
Oct 30 02:46:19.255459 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:19.255402    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:19.292993 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:19.292950    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw" podStartSLOduration=1.8354737970000001 podStartE2EDuration="5.292936203s" totalImagesPullingTime="3.457462406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:46:19.291689896 +0000 UTC m=+324.593868260" watchObservedRunningTime="2026-10-30 02:46:19.292936203 +0000 UTC m=+324.595114568"
Oct 30 02:46:20.243555 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:20.243524    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:20.354920 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:20.354889    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:25.444881 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:46:25.444817    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:46:27.399225 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:27.399188    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:27.571521 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:27.571482    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:30.260720 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:30.260693    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5548b4dc5c-pmdmw"
Oct 30 02:46:44.737352 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:46:44.737315    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:55.444627 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:46:55.444591    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:47:07.131655 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:07.131618    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:13.512783 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:13.512750    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:13.633516 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:13.633484    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:13.755502 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:13.755458    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:13.888418 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:13.888388    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:14.008158 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:14.008128    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:14.284543 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:14.284471    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:16.141868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:16.141793    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:16.252883 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:16.252827    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:16.366903 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:16.366869    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:16.475072 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:16.474991    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:16.584000 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:16.583966    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:25.444159 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:47:25.444124    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:47:30.917322 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:30.917289    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:44.843670 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:47:44.843593    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:55.444630 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:47:55.444583    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:48:05.170556 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:05.170522    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:05.290215 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:05.290182    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:07.585879 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:07.585829    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:14.594247 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:14.594202    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:15.421579 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:15.421547    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:18.412518 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:18.412490    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:18.663813 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:18.663725    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:18.800837 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:18.800803    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:19.408098 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:19.408063    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:21.120353 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:21.120320    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:22.909174 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:22.909143    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:23.836603 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:23.836567    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:25.006232 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:25.006204    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:25.445099 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:48:25.445066    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:48:26.838928 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:26.838895    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:28.381088 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:28.381050    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:28.802588 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:28.802551    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:28.932070 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:28.932039    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:29.432203 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.432167    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"]
Oct 30 02:48:29.435674 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.435654    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"
Oct 30 02:48:29.445790 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.445769    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:29.449208 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.449190    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 02:48:29.449208 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.449190    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 02:48:29.459647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.459624    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 02:48:29.459748 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.459709    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 02:48:29.459748 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.459735    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-88pp5\""
Oct 30 02:48:29.461552 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.461529    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"]
Oct 30 02:48:29.518796 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.518771    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1608971-f5a4-43b3-8a16-e35dd3bc42be-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vhx97\" (UID: \"b1608971-f5a4-43b3-8a16-e35dd3bc42be\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"
Oct 30 02:48:29.518924 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.518815    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5pml\" (UniqueName: \"kubernetes.io/projected/b1608971-f5a4-43b3-8a16-e35dd3bc42be-kube-api-access-h5pml\") pod \"kuadrant-console-plugin-66c4fb575d-vhx97\" (UID: \"b1608971-f5a4-43b3-8a16-e35dd3bc42be\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"
Oct 30 02:48:29.518987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.518962    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b1608971-f5a4-43b3-8a16-e35dd3bc42be-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vhx97\" (UID: \"b1608971-f5a4-43b3-8a16-e35dd3bc42be\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"
Oct 30 02:48:29.569261 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.569235    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:29.620197 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.620175    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1608971-f5a4-43b3-8a16-e35dd3bc42be-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vhx97\" (UID: \"b1608971-f5a4-43b3-8a16-e35dd3bc42be\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"
Oct 30 02:48:29.620294 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.620214    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5pml\" (UniqueName: \"kubernetes.io/projected/b1608971-f5a4-43b3-8a16-e35dd3bc42be-kube-api-access-h5pml\") pod \"kuadrant-console-plugin-66c4fb575d-vhx97\" (UID: \"b1608971-f5a4-43b3-8a16-e35dd3bc42be\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"
Oct 30 02:48:29.620350 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.620337    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b1608971-f5a4-43b3-8a16-e35dd3bc42be-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vhx97\" (UID: \"b1608971-f5a4-43b3-8a16-e35dd3bc42be\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"
Oct 30 02:48:29.620929 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.620911    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b1608971-f5a4-43b3-8a16-e35dd3bc42be-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vhx97\" (UID: \"b1608971-f5a4-43b3-8a16-e35dd3bc42be\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"
Oct 30 02:48:29.622639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.622617    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1608971-f5a4-43b3-8a16-e35dd3bc42be-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vhx97\" (UID: \"b1608971-f5a4-43b3-8a16-e35dd3bc42be\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"
Oct 30 02:48:29.637877 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.637838    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5pml\" (UniqueName: \"kubernetes.io/projected/b1608971-f5a4-43b3-8a16-e35dd3bc42be-kube-api-access-h5pml\") pod \"kuadrant-console-plugin-66c4fb575d-vhx97\" (UID: \"b1608971-f5a4-43b3-8a16-e35dd3bc42be\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"
Oct 30 02:48:29.681192 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.681159    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:29.747303 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:29.747251    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"
Oct 30 02:48:30.040916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:30.040815    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97"]
Oct 30 02:48:30.045712 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:48:30.045686    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1608971_f5a4_43b3_8a16_e35dd3bc42be.slice/crio-54b3ca8f57724ea09786ba3e6123b469516966ea75672c5c2b1fa44ed997931b WatchSource:0}: Error finding container 54b3ca8f57724ea09786ba3e6123b469516966ea75672c5c2b1fa44ed997931b: Status 404 returned error can't find the container with id 54b3ca8f57724ea09786ba3e6123b469516966ea75672c5c2b1fa44ed997931b
Oct 30 02:48:30.781768 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:30.781706    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97" event={"ID":"b1608971-f5a4-43b3-8a16-e35dd3bc42be","Type":"ContainerStarted","Data":"54b3ca8f57724ea09786ba3e6123b469516966ea75672c5c2b1fa44ed997931b"}
Oct 30 02:48:34.803171 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:34.803136    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97" event={"ID":"b1608971-f5a4-43b3-8a16-e35dd3bc42be","Type":"ContainerStarted","Data":"613453de5e01179c85ba99c3443108aff178697f14a06c30a7be9aafdd600f5c"}
Oct 30 02:48:34.808055 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:34.808033    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:34.836497 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:34.836457    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhx97" podStartSLOduration=1.763484678 podStartE2EDuration="5.83644575s" totalImagesPullingTime="4.072961072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:34.835394816 +0000 UTC m=+460.137573181" watchObservedRunningTime="2026-10-30 02:48:34.83644575 +0000 UTC m=+460.138624114"
Oct 30 02:48:34.926484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:34.926459    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:35.402639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:35.402588    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:35.516964 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:35.516932    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:35.999444 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:35.999400    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:36.131006 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:36.130975    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:41.972574 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:41.972541    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:43.034161 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:43.034131    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:43.159600 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:43.159568    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:44.954490 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:44.954455    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:46.041413 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:46.041330    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:46.183001 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:46.182972    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:51.390625 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:51.390589    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:51.517214 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:51.517182    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:52.407263 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:52.407230    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:52.529343 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:52.529313    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:55.444254 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:48:55.444218    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:48:57.040408 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:57.040373    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:58.601894 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:58.601834    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:58.729926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:58.729895    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:59.203645 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:59.203614    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:59.319740 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:48:59.319707    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:00.409691 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:00.409655    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:00.525152 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:00.525118    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:06.204587 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:06.204510    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:06.315279 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:06.315237    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:08.670436 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:08.670405    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:08.779883 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:08.779831    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:11.289951 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:11.289916    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:11.412876 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:11.412827    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:24.985198 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:24.985159    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:25.095442 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:25.095411    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:25.217987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:25.217957    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:25.444993 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:49:25.444956    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:49:25.806393 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:25.806361    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2njfx"]
Oct 30 02:49:25.818398 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:25.818372    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2njfx"
Oct 30 02:49:25.827050 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:25.827026    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2njfx"]
Oct 30 02:49:25.835127 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:25.835110    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s7bx9\""
Oct 30 02:49:25.853178 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:25.853155    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gjc8\" (UniqueName: \"kubernetes.io/projected/983bb4fd-b126-4c5c-87a5-7c304c71adc7-kube-api-access-2gjc8\") pod \"authorino-57f7f8659f-2njfx\" (UID: \"983bb4fd-b126-4c5c-87a5-7c304c71adc7\") " pod="kuadrant-system/authorino-57f7f8659f-2njfx"
Oct 30 02:49:25.908027 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:25.907999    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:25.954185 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:25.954154    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gjc8\" (UniqueName: \"kubernetes.io/projected/983bb4fd-b126-4c5c-87a5-7c304c71adc7-kube-api-access-2gjc8\") pod \"authorino-57f7f8659f-2njfx\" (UID: \"983bb4fd-b126-4c5c-87a5-7c304c71adc7\") " pod="kuadrant-system/authorino-57f7f8659f-2njfx"
Oct 30 02:49:25.971901 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:25.971878    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gjc8\" (UniqueName: \"kubernetes.io/projected/983bb4fd-b126-4c5c-87a5-7c304c71adc7-kube-api-access-2gjc8\") pod \"authorino-57f7f8659f-2njfx\" (UID: \"983bb4fd-b126-4c5c-87a5-7c304c71adc7\") " pod="kuadrant-system/authorino-57f7f8659f-2njfx"
Oct 30 02:49:26.029297 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:26.029271    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:26.130576 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:26.130488    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2njfx"
Oct 30 02:49:26.315254 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:26.315220    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fhvnk"]
Oct 30 02:49:26.318881 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:26.318841    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk"
Oct 30 02:49:26.343139 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:26.343113    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fhvnk"]
Oct 30 02:49:26.357146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:26.357125    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24g8p\" (UniqueName: \"kubernetes.io/projected/2899d418-8570-4b51-a533-59daa56ab9c2-kube-api-access-24g8p\") pod \"authorino-56c6c9ddd5-fhvnk\" (UID: \"2899d418-8570-4b51-a533-59daa56ab9c2\") " pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk"
Oct 30 02:49:26.421578 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:26.421511    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2njfx"]
Oct 30 02:49:26.427903 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:49:26.427879    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod983bb4fd_b126_4c5c_87a5_7c304c71adc7.slice/crio-418bfff462f981cc8dd2a429a966e10f5ca6a3a021f326495ef6abd388af67f5 WatchSource:0}: Error finding container 418bfff462f981cc8dd2a429a966e10f5ca6a3a021f326495ef6abd388af67f5: Status 404 returned error can't find the container with id 418bfff462f981cc8dd2a429a966e10f5ca6a3a021f326495ef6abd388af67f5
Oct 30 02:49:26.458251 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:26.458222    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24g8p\" (UniqueName: \"kubernetes.io/projected/2899d418-8570-4b51-a533-59daa56ab9c2-kube-api-access-24g8p\") pod \"authorino-56c6c9ddd5-fhvnk\" (UID: \"2899d418-8570-4b51-a533-59daa56ab9c2\") " pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk"
Oct 30 02:49:26.475436 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:26.475410    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24g8p\" (UniqueName: \"kubernetes.io/projected/2899d418-8570-4b51-a533-59daa56ab9c2-kube-api-access-24g8p\") pod \"authorino-56c6c9ddd5-fhvnk\" (UID: \"2899d418-8570-4b51-a533-59daa56ab9c2\") " pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk"
Oct 30 02:49:26.630328 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:26.630301    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk"
Oct 30 02:49:26.922492 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:26.922459    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fhvnk"]
Oct 30 02:49:26.927798 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:49:26.927737    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2899d418_8570_4b51_a533_59daa56ab9c2.slice/crio-fcf9eaa7457c13ce569c4f522c698e64b3f8277728285e7fba52aefae3345806 WatchSource:0}: Error finding container fcf9eaa7457c13ce569c4f522c698e64b3f8277728285e7fba52aefae3345806: Status 404 returned error can't find the container with id fcf9eaa7457c13ce569c4f522c698e64b3f8277728285e7fba52aefae3345806
Oct 30 02:49:27.017923 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:27.017895    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk" event={"ID":"2899d418-8570-4b51-a533-59daa56ab9c2","Type":"ContainerStarted","Data":"fcf9eaa7457c13ce569c4f522c698e64b3f8277728285e7fba52aefae3345806"}
Oct 30 02:49:27.019067 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:27.019045    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2njfx" event={"ID":"983bb4fd-b126-4c5c-87a5-7c304c71adc7","Type":"ContainerStarted","Data":"418bfff462f981cc8dd2a429a966e10f5ca6a3a021f326495ef6abd388af67f5"}
Oct 30 02:49:30.037572 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:30.037536    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2njfx" event={"ID":"983bb4fd-b126-4c5c-87a5-7c304c71adc7","Type":"ContainerStarted","Data":"5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7"}
Oct 30 02:49:30.038956 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:30.038933    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk" event={"ID":"2899d418-8570-4b51-a533-59daa56ab9c2","Type":"ContainerStarted","Data":"0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d"}
Oct 30 02:49:30.073363 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:30.073189    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-2njfx" podStartSLOduration=2.240733097 podStartE2EDuration="5.073174825s" totalImagesPullingTime="2.832441728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:30.07193606 +0000 UTC m=+515.374114435" watchObservedRunningTime="2026-10-30 02:49:30.073174825 +0000 UTC m=+515.375353189"
Oct 30 02:49:30.105018 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:30.104971    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk" podStartSLOduration=1.5725570009999998 podStartE2EDuration="4.10495884s" totalImagesPullingTime="2.532401839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:30.104333214 +0000 UTC m=+515.406511582" watchObservedRunningTime="2026-10-30 02:49:30.10495884 +0000 UTC m=+515.407137204"
Oct 30 02:49:30.124267 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:30.124239    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2njfx"]
Oct 30 02:49:32.047136 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:32.047098    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-2njfx" podUID="983bb4fd-b126-4c5c-87a5-7c304c71adc7" containerName="authorino" containerID="cri-o://5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7" gracePeriod=30
Oct 30 02:49:32.286032 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:32.286009    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2njfx"
Oct 30 02:49:32.407304 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:32.407231    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983bb4fd-b126-4c5c-87a5-7c304c71adc7-kube-api-access-2gjc8\" (UniqueName: \"kubernetes.io/projected/983bb4fd-b126-4c5c-87a5-7c304c71adc7-kube-api-access-2gjc8\") pod \"983bb4fd-b126-4c5c-87a5-7c304c71adc7\" (UID: \"983bb4fd-b126-4c5c-87a5-7c304c71adc7\") "
Oct 30 02:49:32.409399 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:32.409370    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983bb4fd-b126-4c5c-87a5-7c304c71adc7-kube-api-access-2gjc8" pod "983bb4fd-b126-4c5c-87a5-7c304c71adc7" (UID: "983bb4fd-b126-4c5c-87a5-7c304c71adc7"). InnerVolumeSpecName "kube-api-access-2gjc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:49:32.508372 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:32.508346    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gjc8\" (UniqueName: \"kubernetes.io/projected/983bb4fd-b126-4c5c-87a5-7c304c71adc7-kube-api-access-2gjc8\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:49:33.052048 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:33.052013    5061 generic.go:356] "Generic (PLEG): container finished" podID="983bb4fd-b126-4c5c-87a5-7c304c71adc7" containerID="5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7" exitCode=0
Oct 30 02:49:33.052421 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:33.052054    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2njfx"
Oct 30 02:49:33.052421 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:33.052098    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2njfx" event={"ID":"983bb4fd-b126-4c5c-87a5-7c304c71adc7","Type":"ContainerDied","Data":"5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7"}
Oct 30 02:49:33.052421 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:33.052133    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2njfx" event={"ID":"983bb4fd-b126-4c5c-87a5-7c304c71adc7","Type":"ContainerDied","Data":"418bfff462f981cc8dd2a429a966e10f5ca6a3a021f326495ef6abd388af67f5"}
Oct 30 02:49:33.052421 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:33.052148    5061 scope.go:122] "RemoveContainer" containerID="5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7"
Oct 30 02:49:33.060139 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:33.060113    5061 scope.go:122] "RemoveContainer" containerID="5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7"
Oct 30 02:49:33.060419 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:49:33.060397    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7\": container with ID starting with 5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7 not found: ID does not exist" containerID="5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7"
Oct 30 02:49:33.060511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:33.060425    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7"} err="failed to get container status \"5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7\": rpc error: code = NotFound desc = could not find container \"5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7\": container with ID starting with 5624edf9ede42d89637d1efb2e2daa24e7193fdf8d60ed339df6b7baa20824d7 not found: ID does not exist"
Oct 30 02:49:33.108387 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:33.108316    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2njfx"]
Oct 30 02:49:33.118613 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:33.118587    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2njfx"]
Oct 30 02:49:34.844973 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:34.844939    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983bb4fd-b126-4c5c-87a5-7c304c71adc7" path="/var/lib/kubelet/pods/983bb4fd-b126-4c5c-87a5-7c304c71adc7/volumes"
Oct 30 02:49:45.123455 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:45.123416    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:46.037530 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:46.037456    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:46.174948 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:46.174917    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:46.873704 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:46.873641    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:55.444417 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:49:55.444376    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:49:58.798323 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:58.798286    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:58.920207 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:58.920173    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:59.063082 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.063002    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2cp4f"]
Oct 30 02:49:59.063332 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.063320    5061 cpu_manager.go:414] "containerMap: removing container" podUID="983bb4fd-b126-4c5c-87a5-7c304c71adc7" containerName="authorino"
Oct 30 02:49:59.063378 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.063333    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983bb4fd-b126-4c5c-87a5-7c304c71adc7" containerName="authorino"
Oct 30 02:49:59.063429 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.063419    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="983bb4fd-b126-4c5c-87a5-7c304c71adc7" containerName="authorino"
Oct 30 02:49:59.067690 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.067669    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2cp4f"
Oct 30 02:49:59.084897 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.084871    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2cp4f"]
Oct 30 02:49:59.214610 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.214575    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlq5x\" (UniqueName: \"kubernetes.io/projected/ade86894-bf3f-4620-ba20-75c810674ed8-kube-api-access-hlq5x\") pod \"authorino-6f6944bcdd-2cp4f\" (UID: \"ade86894-bf3f-4620-ba20-75c810674ed8\") " pod="kuadrant-system/authorino-6f6944bcdd-2cp4f"
Oct 30 02:49:59.259431 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.259398    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2cp4f"]
Oct 30 02:49:59.259612 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:49:59.259595    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hlq5x], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-2cp4f" podUID="ade86894-bf3f-4620-ba20-75c810674ed8"
Oct 30 02:49:59.296681 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.296644    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8q5jl"]
Oct 30 02:49:59.300134 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.300116    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8q5jl"
Oct 30 02:49:59.313003 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.312982    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 02:49:59.315329 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.315289    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlq5x\" (UniqueName: \"kubernetes.io/projected/ade86894-bf3f-4620-ba20-75c810674ed8-kube-api-access-hlq5x\") pod \"authorino-6f6944bcdd-2cp4f\" (UID: \"ade86894-bf3f-4620-ba20-75c810674ed8\") " pod="kuadrant-system/authorino-6f6944bcdd-2cp4f"
Oct 30 02:49:59.318001 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.317973    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8q5jl"]
Oct 30 02:49:59.332896 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.332875    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlq5x\" (UniqueName: \"kubernetes.io/projected/ade86894-bf3f-4620-ba20-75c810674ed8-kube-api-access-hlq5x\") pod \"authorino-6f6944bcdd-2cp4f\" (UID: \"ade86894-bf3f-4620-ba20-75c810674ed8\") " pod="kuadrant-system/authorino-6f6944bcdd-2cp4f"
Oct 30 02:49:59.405586 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.405564    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8q5jl"]
Oct 30 02:49:59.405743 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:49:59.405727    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xbb7p tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-8q5jl" podUID="5d72e3b4-eccb-4f1f-b523-b66e87c87b18"
Oct 30 02:49:59.416637 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.416610    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-tls-cert\") pod \"authorino-6889cbd8c4-8q5jl\" (UID: \"5d72e3b4-eccb-4f1f-b523-b66e87c87b18\") " pod="kuadrant-system/authorino-6889cbd8c4-8q5jl"
Oct 30 02:49:59.416736 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.416692    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbb7p\" (UniqueName: \"kubernetes.io/projected/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-kube-api-access-xbb7p\") pod \"authorino-6889cbd8c4-8q5jl\" (UID: \"5d72e3b4-eccb-4f1f-b523-b66e87c87b18\") " pod="kuadrant-system/authorino-6889cbd8c4-8q5jl"
Oct 30 02:49:59.439170 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.439142    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f7c67bf84-ll94t"]
Oct 30 02:49:59.442441 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.442423    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7c67bf84-ll94t"
Oct 30 02:49:59.466264 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.466245    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f7c67bf84-ll94t"]
Oct 30 02:49:59.517617 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.517593    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbb7p\" (UniqueName: \"kubernetes.io/projected/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-kube-api-access-xbb7p\") pod \"authorino-6889cbd8c4-8q5jl\" (UID: \"5d72e3b4-eccb-4f1f-b523-b66e87c87b18\") " pod="kuadrant-system/authorino-6889cbd8c4-8q5jl"
Oct 30 02:49:59.517711 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.517630    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-tls-cert\") pod \"authorino-6889cbd8c4-8q5jl\" (UID: \"5d72e3b4-eccb-4f1f-b523-b66e87c87b18\") " pod="kuadrant-system/authorino-6889cbd8c4-8q5jl"
Oct 30 02:49:59.519913 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.519894    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-tls-cert\") pod \"authorino-6889cbd8c4-8q5jl\" (UID: \"5d72e3b4-eccb-4f1f-b523-b66e87c87b18\") " pod="kuadrant-system/authorino-6889cbd8c4-8q5jl"
Oct 30 02:49:59.535836 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.535815    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbb7p\" (UniqueName: \"kubernetes.io/projected/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-kube-api-access-xbb7p\") pod \"authorino-6889cbd8c4-8q5jl\" (UID: \"5d72e3b4-eccb-4f1f-b523-b66e87c87b18\") " pod="kuadrant-system/authorino-6889cbd8c4-8q5jl"
Oct 30 02:49:59.618947 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.618889    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqq2\" (UniqueName: \"kubernetes.io/projected/16e510e4-d025-40ae-bbb6-436f11bdf409-kube-api-access-9kqq2\") pod \"authorino-7f7c67bf84-ll94t\" (UID: \"16e510e4-d025-40ae-bbb6-436f11bdf409\") " pod="kuadrant-system/authorino-7f7c67bf84-ll94t"
Oct 30 02:49:59.618947 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.618936    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16e510e4-d025-40ae-bbb6-436f11bdf409-tls-cert\") pod \"authorino-7f7c67bf84-ll94t\" (UID: \"16e510e4-d025-40ae-bbb6-436f11bdf409\") " pod="kuadrant-system/authorino-7f7c67bf84-ll94t"
Oct 30 02:49:59.720263 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.720238    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqq2\" (UniqueName: \"kubernetes.io/projected/16e510e4-d025-40ae-bbb6-436f11bdf409-kube-api-access-9kqq2\") pod \"authorino-7f7c67bf84-ll94t\" (UID: \"16e510e4-d025-40ae-bbb6-436f11bdf409\") " pod="kuadrant-system/authorino-7f7c67bf84-ll94t"
Oct 30 02:49:59.720354 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.720284    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16e510e4-d025-40ae-bbb6-436f11bdf409-tls-cert\") pod \"authorino-7f7c67bf84-ll94t\" (UID: \"16e510e4-d025-40ae-bbb6-436f11bdf409\") " pod="kuadrant-system/authorino-7f7c67bf84-ll94t"
Oct 30 02:49:59.722628 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.722608    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16e510e4-d025-40ae-bbb6-436f11bdf409-tls-cert\") pod \"authorino-7f7c67bf84-ll94t\" (UID: \"16e510e4-d025-40ae-bbb6-436f11bdf409\") " pod="kuadrant-system/authorino-7f7c67bf84-ll94t"
Oct 30 02:49:59.737584 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.737562    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqq2\" (UniqueName: \"kubernetes.io/projected/16e510e4-d025-40ae-bbb6-436f11bdf409-kube-api-access-9kqq2\") pod \"authorino-7f7c67bf84-ll94t\" (UID: \"16e510e4-d025-40ae-bbb6-436f11bdf409\") " pod="kuadrant-system/authorino-7f7c67bf84-ll94t"
Oct 30 02:49:59.753330 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:49:59.753313    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7c67bf84-ll94t"
Oct 30 02:50:00.046770 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.046738    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f7c67bf84-ll94t"]
Oct 30 02:50:00.051519 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:50:00.051493    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16e510e4_d025_40ae_bbb6_436f11bdf409.slice/crio-663ba5f0523b28629411b249422b6eb4f81dbe5256f37abffa3a9f8018f48710 WatchSource:0}: Error finding container 663ba5f0523b28629411b249422b6eb4f81dbe5256f37abffa3a9f8018f48710: Status 404 returned error can't find the container with id 663ba5f0523b28629411b249422b6eb4f81dbe5256f37abffa3a9f8018f48710
Oct 30 02:50:00.164772 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.164737    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f7c67bf84-ll94t" event={"ID":"16e510e4-d025-40ae-bbb6-436f11bdf409","Type":"ContainerStarted","Data":"663ba5f0523b28629411b249422b6eb4f81dbe5256f37abffa3a9f8018f48710"}
Oct 30 02:50:00.164772 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.164762    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8q5jl"
Oct 30 02:50:00.164989 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.164762    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2cp4f"
Oct 30 02:50:00.173002 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.172981    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8q5jl"
Oct 30 02:50:00.175018 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.175003    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2cp4f"
Oct 30 02:50:00.324981 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.324911    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-tls-cert\" (UniqueName: \"kubernetes.io/secret/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-tls-cert\") pod \"5d72e3b4-eccb-4f1f-b523-b66e87c87b18\" (UID: \"5d72e3b4-eccb-4f1f-b523-b66e87c87b18\") "
Oct 30 02:50:00.324981 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.324978    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-kube-api-access-xbb7p\" (UniqueName: \"kubernetes.io/projected/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-kube-api-access-xbb7p\") pod \"5d72e3b4-eccb-4f1f-b523-b66e87c87b18\" (UID: \"5d72e3b4-eccb-4f1f-b523-b66e87c87b18\") "
Oct 30 02:50:00.325145 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.325023    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade86894-bf3f-4620-ba20-75c810674ed8-kube-api-access-hlq5x\" (UniqueName: \"kubernetes.io/projected/ade86894-bf3f-4620-ba20-75c810674ed8-kube-api-access-hlq5x\") pod \"ade86894-bf3f-4620-ba20-75c810674ed8\" (UID: \"ade86894-bf3f-4620-ba20-75c810674ed8\") "
Oct 30 02:50:00.327208 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.327181    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-tls-cert" pod "5d72e3b4-eccb-4f1f-b523-b66e87c87b18" (UID: "5d72e3b4-eccb-4f1f-b523-b66e87c87b18"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:50:00.327309 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.327223    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade86894-bf3f-4620-ba20-75c810674ed8-kube-api-access-hlq5x" pod "ade86894-bf3f-4620-ba20-75c810674ed8" (UID: "ade86894-bf3f-4620-ba20-75c810674ed8"). InnerVolumeSpecName "kube-api-access-hlq5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:50:00.327309 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.327242    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-kube-api-access-xbb7p" pod "5d72e3b4-eccb-4f1f-b523-b66e87c87b18" (UID: "5d72e3b4-eccb-4f1f-b523-b66e87c87b18"). InnerVolumeSpecName "kube-api-access-xbb7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:50:00.425988 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.425967    5061 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-tls-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:50:00.426077 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.425988    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbb7p\" (UniqueName: \"kubernetes.io/projected/5d72e3b4-eccb-4f1f-b523-b66e87c87b18-kube-api-access-xbb7p\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:50:00.426077 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:00.425999    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlq5x\" (UniqueName: \"kubernetes.io/projected/ade86894-bf3f-4620-ba20-75c810674ed8-kube-api-access-hlq5x\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:50:01.170564 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.170475    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2cp4f"
Oct 30 02:50:01.171051 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.170475    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f7c67bf84-ll94t" event={"ID":"16e510e4-d025-40ae-bbb6-436f11bdf409","Type":"ContainerStarted","Data":"6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77"}
Oct 30 02:50:01.171051 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.170650    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8q5jl"
Oct 30 02:50:01.207049 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.207000    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7f7c67bf84-ll94t" podStartSLOduration=1.871899042 podStartE2EDuration="2.206988427s" totalImagesPullingTime="335.089385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:50:01.206286123 +0000 UTC m=+546.508464470" watchObservedRunningTime="2026-10-30 02:50:01.206988427 +0000 UTC m=+546.509166791"
Oct 30 02:50:01.234275 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.234249    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fhvnk"]
Oct 30 02:50:01.234434 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.234415    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk" podUID="2899d418-8570-4b51-a533-59daa56ab9c2" containerName="authorino" containerID="cri-o://0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d" gracePeriod=30
Oct 30 02:50:01.272338 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.272314    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2cp4f"]
Oct 30 02:50:01.274836 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.274810    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2cp4f"]
Oct 30 02:50:01.326621 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.326595    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8q5jl"]
Oct 30 02:50:01.331554 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.331528    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8q5jl"]
Oct 30 02:50:01.484315 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.484295    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk"
Oct 30 02:50:01.635443 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.635413    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2899d418-8570-4b51-a533-59daa56ab9c2-kube-api-access-24g8p\" (UniqueName: \"kubernetes.io/projected/2899d418-8570-4b51-a533-59daa56ab9c2-kube-api-access-24g8p\") pod \"2899d418-8570-4b51-a533-59daa56ab9c2\" (UID: \"2899d418-8570-4b51-a533-59daa56ab9c2\") "
Oct 30 02:50:01.637405 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.637379    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2899d418-8570-4b51-a533-59daa56ab9c2-kube-api-access-24g8p" pod "2899d418-8570-4b51-a533-59daa56ab9c2" (UID: "2899d418-8570-4b51-a533-59daa56ab9c2"). InnerVolumeSpecName "kube-api-access-24g8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:50:01.736103 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:01.736041    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24g8p\" (UniqueName: \"kubernetes.io/projected/2899d418-8570-4b51-a533-59daa56ab9c2-kube-api-access-24g8p\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:50:02.176181 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.176146    5061 generic.go:356] "Generic (PLEG): container finished" podID="2899d418-8570-4b51-a533-59daa56ab9c2" containerID="0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d" exitCode=0
Oct 30 02:50:02.176590 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.176185    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk"
Oct 30 02:50:02.176590 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.176222    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk" event={"ID":"2899d418-8570-4b51-a533-59daa56ab9c2","Type":"ContainerDied","Data":"0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d"}
Oct 30 02:50:02.176590 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.176257    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fhvnk" event={"ID":"2899d418-8570-4b51-a533-59daa56ab9c2","Type":"ContainerDied","Data":"fcf9eaa7457c13ce569c4f522c698e64b3f8277728285e7fba52aefae3345806"}
Oct 30 02:50:02.176590 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.176279    5061 scope.go:122] "RemoveContainer" containerID="0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d"
Oct 30 02:50:02.184459 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.184434    5061 scope.go:122] "RemoveContainer" containerID="0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d"
Oct 30 02:50:02.184727 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:50:02.184709    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d\": container with ID starting with 0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d not found: ID does not exist" containerID="0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d"
Oct 30 02:50:02.184783 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.184735    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d"} err="failed to get container status \"0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d\": rpc error: code = NotFound desc = could not find container \"0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d\": container with ID starting with 0de5e58f133d919326cb130149a718a8d398de64acddc5a944cb09d730a5502d not found: ID does not exist"
Oct 30 02:50:02.223308 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.223285    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fhvnk"]
Oct 30 02:50:02.227278 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.227252    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fhvnk"]
Oct 30 02:50:02.844568 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.844535    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2899d418-8570-4b51-a533-59daa56ab9c2" path="/var/lib/kubelet/pods/2899d418-8570-4b51-a533-59daa56ab9c2/volumes"
Oct 30 02:50:02.844928 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.844913    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d72e3b4-eccb-4f1f-b523-b66e87c87b18" path="/var/lib/kubelet/pods/5d72e3b4-eccb-4f1f-b523-b66e87c87b18/volumes"
Oct 30 02:50:02.845121 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:02.845110    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade86894-bf3f-4620-ba20-75c810674ed8" path="/var/lib/kubelet/pods/ade86894-bf3f-4620-ba20-75c810674ed8/volumes"
Oct 30 02:50:08.427522 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:08.427487    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:08.542892 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:08.542821    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:08.654260 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:08.654231    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:08.763607 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:08.763538    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:12.266625 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:12.266588    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:12.418079 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:12.418045    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:15.767191 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:15.767159    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:15.880635 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:15.880601    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:20.040692 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:20.040651    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:25.444398 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:50:25.444359    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:50:44.449305 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:44.449228    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:44.595395 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:44.595337    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:45.213371 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:45.213336    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:45.764548 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:45.764518    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:45.881128 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:45.881096    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:45.998106 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:45.998040    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"]
Oct 30 02:50:45.998393 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:45.998380    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2899d418-8570-4b51-a533-59daa56ab9c2" containerName="authorino"
Oct 30 02:50:45.998439 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:45.998394    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2899d418-8570-4b51-a533-59daa56ab9c2" containerName="authorino"
Oct 30 02:50:45.998518 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:45.998456    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2899d418-8570-4b51-a533-59daa56ab9c2" containerName="authorino"
Oct 30 02:50:46.001577 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.001552    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.011964 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.011945    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:46.026286 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.026238    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 02:50:46.026391 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.026316    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 02:50:46.026391 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.026312    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bzc5m\""
Oct 30 02:50:46.026391 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.026373    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 02:50:46.026513 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.026445    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 02:50:46.038024 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.038003    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"]
Oct 30 02:50:46.069448 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.069424    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d499f-545e-4012-a139-cb0268bef829-openshift-service-ca\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.069538 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.069451    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69k8h\" (UniqueName: \"kubernetes.io/projected/0f6d499f-545e-4012-a139-cb0268bef829-kube-api-access-69k8h\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.069538 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.069480    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-metrics-certs\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.069607 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.069591    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-maas-api-tls\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.170581 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.170557    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-maas-api-tls\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.170678 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.170602    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d499f-545e-4012-a139-cb0268bef829-openshift-service-ca\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.170678 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.170621    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69k8h\" (UniqueName: \"kubernetes.io/projected/0f6d499f-545e-4012-a139-cb0268bef829-kube-api-access-69k8h\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.170678 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.170646    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-metrics-certs\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.171236 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.171216    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d499f-545e-4012-a139-cb0268bef829-openshift-service-ca\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.173065 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.173042    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-maas-api-tls\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.173147 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.173090    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-metrics-certs\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.187364 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.187342    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69k8h\" (UniqueName: \"kubernetes.io/projected/0f6d499f-545e-4012-a139-cb0268bef829-kube-api-access-69k8h\") pod \"maas-api-5c48877c75-lpjs6\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.313523 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.313500    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:46.606676 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:46.606605    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"]
Oct 30 02:50:46.611614 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:50:46.611583    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6d499f_545e_4012_a139_cb0268bef829.slice/crio-0437b72576e15758a123cea70e3bd316ad1fcc45356dccac39886112bf8d4093 WatchSource:0}: Error finding container 0437b72576e15758a123cea70e3bd316ad1fcc45356dccac39886112bf8d4093: Status 404 returned error can't find the container with id 0437b72576e15758a123cea70e3bd316ad1fcc45356dccac39886112bf8d4093
Oct 30 02:50:47.376307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:47.376265    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6" event={"ID":"0f6d499f-545e-4012-a139-cb0268bef829","Type":"ContainerStarted","Data":"0437b72576e15758a123cea70e3bd316ad1fcc45356dccac39886112bf8d4093"}
Oct 30 02:50:49.387616 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:49.387579    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6" event={"ID":"0f6d499f-545e-4012-a139-cb0268bef829","Type":"ContainerStarted","Data":"b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0"}
Oct 30 02:50:49.388073 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:49.387668    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:49.422327 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:49.422281    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6" podStartSLOduration=2.095003502 podStartE2EDuration="4.422271321s" totalImagesPullingTime="2.327267819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:50:49.421664563 +0000 UTC m=+594.723842932" watchObservedRunningTime="2026-10-30 02:50:49.422271321 +0000 UTC m=+594.724449686"
Oct 30 02:50:54.892110 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:54.891998    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:50:54.894028 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:54.894001    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:50:54.895301 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:54.895281    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:50:54.897331 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:54.897308    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:50:55.397934 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:50:55.397909    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 02:50:55.446371 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:50:55.446317    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:51:25.444895 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:51:25.444860    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:51:29.812035 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:29.812000    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:51:29.958205 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:29.958138    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:51:45.307026 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:45.306982    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:51:49.407689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:49.407650    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:51:51.614999 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.614965    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"]
Oct 30 02:51:51.618459 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.618437    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.632629 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.632608    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 02:51:51.632763 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.632683    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 02:51:51.643210 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.643184    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9xfvj\""
Oct 30 02:51:51.643384 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.643363    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 02:51:51.644734 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.644707    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"]
Oct 30 02:51:51.688516 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.688481    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.688647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.688585    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.688647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.688629    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.688751 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.688727    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbtpf\" (UniqueName: \"kubernetes.io/projected/e686dfeb-afd4-4de3-8528-60bb29527f1e-kube-api-access-qbtpf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.688787 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.688773    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e686dfeb-afd4-4de3-8528-60bb29527f1e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.688870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.688841    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.717267 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.717229    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:51:51.789293 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.789253    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.789467 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.789298    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtpf\" (UniqueName: \"kubernetes.io/projected/e686dfeb-afd4-4de3-8528-60bb29527f1e-kube-api-access-qbtpf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.789467 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.789336    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e686dfeb-afd4-4de3-8528-60bb29527f1e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.789467 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.789377    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.789467 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.789432    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.789695 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.789492    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.789910 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.789878    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.790045 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.789928    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.790045 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.789983    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.791888 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.791869    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e686dfeb-afd4-4de3-8528-60bb29527f1e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.791988 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.791900    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e686dfeb-afd4-4de3-8528-60bb29527f1e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.806357 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.806335    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtpf\" (UniqueName: \"kubernetes.io/projected/e686dfeb-afd4-4de3-8528-60bb29527f1e-kube-api-access-qbtpf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-67pfh\" (UID: \"e686dfeb-afd4-4de3-8528-60bb29527f1e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:51.932668 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:51.932566    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:52.231146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:52.231073    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"]
Oct 30 02:51:52.236239 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:51:52.236215    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode686dfeb_afd4_4de3_8528_60bb29527f1e.slice/crio-ed559a6390b96c872fc6e63cc3ce129793c1ce810d595f469faa54b0ac860bbc WatchSource:0}: Error finding container ed559a6390b96c872fc6e63cc3ce129793c1ce810d595f469faa54b0ac860bbc: Status 404 returned error can't find the container with id ed559a6390b96c872fc6e63cc3ce129793c1ce810d595f469faa54b0ac860bbc
Oct 30 02:51:52.237533 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:52.237518    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:51:52.656315 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:52.656255    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh" event={"ID":"e686dfeb-afd4-4de3-8528-60bb29527f1e","Type":"ContainerStarted","Data":"ed559a6390b96c872fc6e63cc3ce129793c1ce810d595f469faa54b0ac860bbc"}
Oct 30 02:51:53.661671 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:53.661593    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh" event={"ID":"e686dfeb-afd4-4de3-8528-60bb29527f1e","Type":"ContainerStarted","Data":"752f7e3232df3670806b7b1a49c2851ab67c73fc49d7e4e5110f7845120215a4"}
Oct 30 02:51:53.699116 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:53.699069    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh" podStartSLOduration=1.587713031 podStartE2EDuration="2.699055831s" totalImagesPullingTime="1.1113428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:51:53.698063446 +0000 UTC m=+659.000241810" watchObservedRunningTime="2026-10-30 02:51:53.699055831 +0000 UTC m=+659.001234195"
Oct 30 02:51:53.932999 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:53.932902    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:51:55.239817 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:55.239785    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:51:55.445594 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:51:55.445559    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:51:56.267297 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.267261    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"]
Oct 30 02:51:56.271534 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.271508    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.292339 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.292316    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"]
Oct 30 02:51:56.297508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.297483    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 02:51:56.342238 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.342204    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:51:56.448868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.448243    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b66266-1f25-4615-984e-ebd9fbfb5cac-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.448868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.448291    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbc8\" (UniqueName: \"kubernetes.io/projected/47b66266-1f25-4615-984e-ebd9fbfb5cac-kube-api-access-9mbc8\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.448868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.448371    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.448868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.448395    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.448868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.448423    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.448868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.448483    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.549463 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.549431    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbc8\" (UniqueName: \"kubernetes.io/projected/47b66266-1f25-4615-984e-ebd9fbfb5cac-kube-api-access-9mbc8\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.549651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.549486    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.549651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.549506    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.549651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.549528    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.549651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.549585    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.549651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.549624    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b66266-1f25-4615-984e-ebd9fbfb5cac-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.549992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.549959    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.550068 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.550007    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.550141 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.550118    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.551883 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.551834    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47b66266-1f25-4615-984e-ebd9fbfb5cac-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.552260 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.552240    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b66266-1f25-4615-984e-ebd9fbfb5cac-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.567209 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.567179    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbc8\" (UniqueName: \"kubernetes.io/projected/47b66266-1f25-4615-984e-ebd9fbfb5cac-kube-api-access-9mbc8\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq\" (UID: \"47b66266-1f25-4615-984e-ebd9fbfb5cac\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.585002 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.584977    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:56.881056 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:56.880982    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"]
Oct 30 02:51:56.887079 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:51:56.887049    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b66266_1f25_4615_984e_ebd9fbfb5cac.slice/crio-c02daa756cb6b843624538c72a4a7aa334bc2a43d5321b650860a2923746c800 WatchSource:0}: Error finding container c02daa756cb6b843624538c72a4a7aa334bc2a43d5321b650860a2923746c800: Status 404 returned error can't find the container with id c02daa756cb6b843624538c72a4a7aa334bc2a43d5321b650860a2923746c800
Oct 30 02:51:57.680862 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:57.680813    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq" event={"ID":"47b66266-1f25-4615-984e-ebd9fbfb5cac","Type":"ContainerStarted","Data":"0064b10ff414981e8f9f6c64ef67c7af36fbdbf56b3a7092e971a4a52eab89db"}
Oct 30 02:51:57.681221 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:57.680903    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq" event={"ID":"47b66266-1f25-4615-984e-ebd9fbfb5cac","Type":"ContainerStarted","Data":"c02daa756cb6b843624538c72a4a7aa334bc2a43d5321b650860a2923746c800"}
Oct 30 02:51:57.716741 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:57.716694    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq" podStartSLOduration=1.5358175790000002 podStartE2EDuration="1.716683796s" totalImagesPullingTime="180.866217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:51:57.715810979 +0000 UTC m=+663.017989344" watchObservedRunningTime="2026-10-30 02:51:57.716683796 +0000 UTC m=+663.018862160"
Oct 30 02:51:58.585671 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:58.585627    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:51:59.099790 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.099751    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"]
Oct 30 02:51:59.103818 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.103797    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.129536 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.129501    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 02:51:59.130945 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.130925    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"]
Oct 30 02:51:59.191479 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.191446    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:51:59.272036 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.272008    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.272160 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.272042    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.272160 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.272068    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdjb\" (UniqueName: \"kubernetes.io/projected/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-kube-api-access-hxdjb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.272230 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.272175    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.272230 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.272221    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.272294 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.272279    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.373577 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.373504    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.373577 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.373537    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.373577 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.373562    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.373789 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.373595    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.373789 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.373617    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.373789 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.373638    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdjb\" (UniqueName: \"kubernetes.io/projected/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-kube-api-access-hxdjb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.374083 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.374061    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.374159 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.374102    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.374159 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.374124    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.375995 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.375976    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.376068 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.376051    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.390567 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.390548    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdjb\" (UniqueName: \"kubernetes.io/projected/24ea5b2c-abf7-4857-86c5-8c1690ee76f0-kube-api-access-hxdjb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf\" (UID: \"24ea5b2c-abf7-4857-86c5-8c1690ee76f0\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.415959 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.415937    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:51:59.708498 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:51:59.708395    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"]
Oct 30 02:51:59.713026 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:51:59.713001    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24ea5b2c_abf7_4857_86c5_8c1690ee76f0.slice/crio-263a9c6d24a8186d016971dae32c3a8cd380385e2a9699eba5a90fdf2196b0c9 WatchSource:0}: Error finding container 263a9c6d24a8186d016971dae32c3a8cd380385e2a9699eba5a90fdf2196b0c9: Status 404 returned error can't find the container with id 263a9c6d24a8186d016971dae32c3a8cd380385e2a9699eba5a90fdf2196b0c9
Oct 30 02:52:00.547519 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:00.547477    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:00.695006 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:00.694972    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" event={"ID":"24ea5b2c-abf7-4857-86c5-8c1690ee76f0","Type":"ContainerStarted","Data":"69a92f122eef4a0992fab6b14361b4ac48ebc1d628162794a914cd780263476d"}
Oct 30 02:52:00.695006 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:00.695013    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" event={"ID":"24ea5b2c-abf7-4857-86c5-8c1690ee76f0","Type":"ContainerStarted","Data":"263a9c6d24a8186d016971dae32c3a8cd380385e2a9699eba5a90fdf2196b0c9"}
Oct 30 02:52:00.730894 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:00.730826    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podStartSLOduration=1.536020786 podStartE2EDuration="1.730813555s" totalImagesPullingTime="194.792769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:00.730389344 +0000 UTC m=+666.032567708" watchObservedRunningTime="2026-10-30 02:52:00.730813555 +0000 UTC m=+666.032991920"
Oct 30 02:52:01.416955 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:01.416922    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:52:01.933511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:01.933478    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:52:01.945921 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:01.945894    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:52:02.715667 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:02.715639    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-67pfh"
Oct 30 02:52:02.722513 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:02.722473    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:06.585388 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:06.585316    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:52:06.597461 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:06.597438    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:52:06.734069 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:06.734038    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq"
Oct 30 02:52:09.416541 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:09.416504    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:52:09.429262 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:09.429240    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:52:09.747004 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:09.746921    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:52:12.091899 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.091866    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dc464c95f-jdm9p"]
Oct 30 02:52:12.095467 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.095447    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dc464c95f-jdm9p"
Oct 30 02:52:12.113472 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.113452    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dc464c95f-jdm9p"]
Oct 30 02:52:12.277697 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.277665    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/516f14ce-326d-4d8e-a251-6f1466249209-tls-cert\") pod \"authorino-6dc464c95f-jdm9p\" (UID: \"516f14ce-326d-4d8e-a251-6f1466249209\") " pod="kuadrant-system/authorino-6dc464c95f-jdm9p"
Oct 30 02:52:12.277874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.277735    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9lqt\" (UniqueName: \"kubernetes.io/projected/516f14ce-326d-4d8e-a251-6f1466249209-kube-api-access-l9lqt\") pod \"authorino-6dc464c95f-jdm9p\" (UID: \"516f14ce-326d-4d8e-a251-6f1466249209\") " pod="kuadrant-system/authorino-6dc464c95f-jdm9p"
Oct 30 02:52:12.378922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.378821    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9lqt\" (UniqueName: \"kubernetes.io/projected/516f14ce-326d-4d8e-a251-6f1466249209-kube-api-access-l9lqt\") pod \"authorino-6dc464c95f-jdm9p\" (UID: \"516f14ce-326d-4d8e-a251-6f1466249209\") " pod="kuadrant-system/authorino-6dc464c95f-jdm9p"
Oct 30 02:52:12.379065 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.378934    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/516f14ce-326d-4d8e-a251-6f1466249209-tls-cert\") pod \"authorino-6dc464c95f-jdm9p\" (UID: \"516f14ce-326d-4d8e-a251-6f1466249209\") " pod="kuadrant-system/authorino-6dc464c95f-jdm9p"
Oct 30 02:52:12.381295 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.381268    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/516f14ce-326d-4d8e-a251-6f1466249209-tls-cert\") pod \"authorino-6dc464c95f-jdm9p\" (UID: \"516f14ce-326d-4d8e-a251-6f1466249209\") " pod="kuadrant-system/authorino-6dc464c95f-jdm9p"
Oct 30 02:52:12.396105 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.396080    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9lqt\" (UniqueName: \"kubernetes.io/projected/516f14ce-326d-4d8e-a251-6f1466249209-kube-api-access-l9lqt\") pod \"authorino-6dc464c95f-jdm9p\" (UID: \"516f14ce-326d-4d8e-a251-6f1466249209\") " pod="kuadrant-system/authorino-6dc464c95f-jdm9p"
Oct 30 02:52:12.407369 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.407350    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dc464c95f-jdm9p"
Oct 30 02:52:12.569522 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.569493    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:12.683724 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.683661    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:12.698638 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.698610    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dc464c95f-jdm9p"]
Oct 30 02:52:12.703154 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:52:12.703130    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod516f14ce_326d_4d8e_a251_6f1466249209.slice/crio-55dfa9f0ffdc5b08ceac6e2f6ffd6d4331b024ce4958dfa698ba885335094c58 WatchSource:0}: Error finding container 55dfa9f0ffdc5b08ceac6e2f6ffd6d4331b024ce4958dfa698ba885335094c58: Status 404 returned error can't find the container with id 55dfa9f0ffdc5b08ceac6e2f6ffd6d4331b024ce4958dfa698ba885335094c58
Oct 30 02:52:12.751619 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:12.751594    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dc464c95f-jdm9p" event={"ID":"516f14ce-326d-4d8e-a251-6f1466249209","Type":"ContainerStarted","Data":"55dfa9f0ffdc5b08ceac6e2f6ffd6d4331b024ce4958dfa698ba885335094c58"}
Oct 30 02:52:13.759073 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:13.759039    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dc464c95f-jdm9p" event={"ID":"516f14ce-326d-4d8e-a251-6f1466249209","Type":"ContainerStarted","Data":"5ad58e7c97ac9da164ed412306013c166febfc37e6032af376ae958d90a3f66c"}
Oct 30 02:52:13.795039 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:13.794984    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6dc464c95f-jdm9p" podStartSLOduration=0.967088883 podStartE2EDuration="1.794968249s" totalImagesPullingTime="827.879366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:13.792549217 +0000 UTC m=+679.094727582" watchObservedRunningTime="2026-10-30 02:52:13.794968249 +0000 UTC m=+679.097146615"
Oct 30 02:52:13.823121 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:13.823090    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f7c67bf84-ll94t"]
Oct 30 02:52:13.823537 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:13.823499    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7f7c67bf84-ll94t" podUID="16e510e4-d025-40ae-bbb6-436f11bdf409" containerName="authorino" containerID="cri-o://6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77" gracePeriod=30
Oct 30 02:52:14.076442 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.076418    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7c67bf84-ll94t"
Oct 30 02:52:14.093410 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.093376    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e510e4-d025-40ae-bbb6-436f11bdf409-tls-cert\" (UniqueName: \"kubernetes.io/secret/16e510e4-d025-40ae-bbb6-436f11bdf409-tls-cert\") pod \"16e510e4-d025-40ae-bbb6-436f11bdf409\" (UID: \"16e510e4-d025-40ae-bbb6-436f11bdf409\") "
Oct 30 02:52:14.093543 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.093438    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e510e4-d025-40ae-bbb6-436f11bdf409-kube-api-access-9kqq2\" (UniqueName: \"kubernetes.io/projected/16e510e4-d025-40ae-bbb6-436f11bdf409-kube-api-access-9kqq2\") pod \"16e510e4-d025-40ae-bbb6-436f11bdf409\" (UID: \"16e510e4-d025-40ae-bbb6-436f11bdf409\") "
Oct 30 02:52:14.095658 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.095627    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e510e4-d025-40ae-bbb6-436f11bdf409-kube-api-access-9kqq2" pod "16e510e4-d025-40ae-bbb6-436f11bdf409" (UID: "16e510e4-d025-40ae-bbb6-436f11bdf409"). InnerVolumeSpecName "kube-api-access-9kqq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:52:14.108301 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.108268    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e510e4-d025-40ae-bbb6-436f11bdf409-tls-cert" pod "16e510e4-d025-40ae-bbb6-436f11bdf409" (UID: "16e510e4-d025-40ae-bbb6-436f11bdf409"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:52:14.194111 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.194080    5061 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16e510e4-d025-40ae-bbb6-436f11bdf409-tls-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:52:14.194111 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.194106    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kqq2\" (UniqueName: \"kubernetes.io/projected/16e510e4-d025-40ae-bbb6-436f11bdf409-kube-api-access-9kqq2\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:52:14.765839 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.765800    5061 generic.go:356] "Generic (PLEG): container finished" podID="16e510e4-d025-40ae-bbb6-436f11bdf409" containerID="6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77" exitCode=0
Oct 30 02:52:14.766311 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.765875    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7c67bf84-ll94t"
Oct 30 02:52:14.766311 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.765894    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f7c67bf84-ll94t" event={"ID":"16e510e4-d025-40ae-bbb6-436f11bdf409","Type":"ContainerDied","Data":"6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77"}
Oct 30 02:52:14.766311 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.765967    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f7c67bf84-ll94t" event={"ID":"16e510e4-d025-40ae-bbb6-436f11bdf409","Type":"ContainerDied","Data":"663ba5f0523b28629411b249422b6eb4f81dbe5256f37abffa3a9f8018f48710"}
Oct 30 02:52:14.766311 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.766018    5061 scope.go:122] "RemoveContainer" containerID="6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77"
Oct 30 02:52:14.776133 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.776116    5061 scope.go:122] "RemoveContainer" containerID="6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77"
Oct 30 02:52:14.776401 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:52:14.776380    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77\": container with ID starting with 6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77 not found: ID does not exist" containerID="6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77"
Oct 30 02:52:14.776474 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.776411    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77"} err="failed to get container status \"6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77\": rpc error: code = NotFound desc = could not find container \"6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77\": container with ID starting with 6c4830546b8749432a9b279960f08a48aaa5953fc2cb6ec70b7439b86ea87d77 not found: ID does not exist"
Oct 30 02:52:14.809634 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.809607    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f7c67bf84-ll94t"]
Oct 30 02:52:14.820616 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.820587    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f7c67bf84-ll94t"]
Oct 30 02:52:14.845742 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:14.845709    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e510e4-d025-40ae-bbb6-436f11bdf409" path="/var/lib/kubelet/pods/16e510e4-d025-40ae-bbb6-436f11bdf409/volumes"
Oct 30 02:52:16.906429 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:16.906392    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:16.979735 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:16.979700    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-v8fgz"]
Oct 30 02:52:16.980162 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:16.980143    5061 cpu_manager.go:414] "containerMap: removing container" podUID="16e510e4-d025-40ae-bbb6-436f11bdf409" containerName="authorino"
Oct 30 02:52:16.980275 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:16.980163    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e510e4-d025-40ae-bbb6-436f11bdf409" containerName="authorino"
Oct 30 02:52:16.980275 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:16.980241    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e510e4-d025-40ae-bbb6-436f11bdf409" containerName="authorino"
Oct 30 02:52:16.983158 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:16.983136    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz"
Oct 30 02:52:17.017314 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.017276    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87148799-07e3-41d4-a6f0-0e4678d90202-plugins-config-volume\") pod \"payload-processing-5f78687d5b-v8fgz\" (UID: \"87148799-07e3-41d4-a6f0-0e4678d90202\") " pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz"
Oct 30 02:52:17.017454 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.017328    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvbv\" (UniqueName: \"kubernetes.io/projected/87148799-07e3-41d4-a6f0-0e4678d90202-kube-api-access-9cvbv\") pod \"payload-processing-5f78687d5b-v8fgz\" (UID: \"87148799-07e3-41d4-a6f0-0e4678d90202\") " pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz"
Oct 30 02:52:17.036000 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.035972    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-v8fgz"]
Oct 30 02:52:17.087603 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.087574    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 02:52:17.118159 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.118123    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87148799-07e3-41d4-a6f0-0e4678d90202-plugins-config-volume\") pod \"payload-processing-5f78687d5b-v8fgz\" (UID: \"87148799-07e3-41d4-a6f0-0e4678d90202\") " pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz"
Oct 30 02:52:17.118352 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.118188    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvbv\" (UniqueName: \"kubernetes.io/projected/87148799-07e3-41d4-a6f0-0e4678d90202-kube-api-access-9cvbv\") pod \"payload-processing-5f78687d5b-v8fgz\" (UID: \"87148799-07e3-41d4-a6f0-0e4678d90202\") " pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz"
Oct 30 02:52:17.118929 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.118906    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87148799-07e3-41d4-a6f0-0e4678d90202-plugins-config-volume\") pod \"payload-processing-5f78687d5b-v8fgz\" (UID: \"87148799-07e3-41d4-a6f0-0e4678d90202\") " pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz"
Oct 30 02:52:17.144265 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.144230    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:17.146114 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.146082    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvbv\" (UniqueName: \"kubernetes.io/projected/87148799-07e3-41d4-a6f0-0e4678d90202-kube-api-access-9cvbv\") pod \"payload-processing-5f78687d5b-v8fgz\" (UID: \"87148799-07e3-41d4-a6f0-0e4678d90202\") " pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz"
Oct 30 02:52:17.296556 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.296521    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz"
Oct 30 02:52:17.598821 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.598725    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-v8fgz"]
Oct 30 02:52:17.602724 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:52:17.602695    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87148799_07e3_41d4_a6f0_0e4678d90202.slice/crio-0ee3d00aad0850366a1569ed549452ab5c0bc6d2932de3470cf54e6b2f7c0776 WatchSource:0}: Error finding container 0ee3d00aad0850366a1569ed549452ab5c0bc6d2932de3470cf54e6b2f7c0776: Status 404 returned error can't find the container with id 0ee3d00aad0850366a1569ed549452ab5c0bc6d2932de3470cf54e6b2f7c0776
Oct 30 02:52:17.782794 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:17.782757    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz" event={"ID":"87148799-07e3-41d4-a6f0-0e4678d90202","Type":"ContainerStarted","Data":"0ee3d00aad0850366a1569ed549452ab5c0bc6d2932de3470cf54e6b2f7c0776"}
Oct 30 02:52:19.793395 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:19.793349    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz" event={"ID":"87148799-07e3-41d4-a6f0-0e4678d90202","Type":"ContainerStarted","Data":"f6b4d22d381ac614251d2753f154b06e5f3c5994b1d3877aa3cf31c02f61b0cd"}
Oct 30 02:52:19.793804 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:19.793410    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz"
Oct 30 02:52:19.829450 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:19.829401    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz" podStartSLOduration=1.829770493 podStartE2EDuration="3.829387538s" totalImagesPullingTime="1.999617045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:19.828429651 +0000 UTC m=+685.130608016" watchObservedRunningTime="2026-10-30 02:52:19.829387538 +0000 UTC m=+685.131565903"
Oct 30 02:52:25.444832 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:52:25.444792    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:52:26.951930 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:26.951899    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"]
Oct 30 02:52:26.955496 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:26.955475    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:26.976057 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:26.976035    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"]
Oct 30 02:52:26.979304 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:26.979287    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:26.985267 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:26.985245    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 02:52:26.985365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:26.985283    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wg9w9\""
Oct 30 02:52:26.987428 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:26.987408    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"]
Oct 30 02:52:27.012266 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.012234    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.012432 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.012285    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.012432 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.012329    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.012432 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.012364    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.012432 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.012397    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmdn\" (UniqueName: \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-kube-api-access-vvmdn\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.012432 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.012441    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.013131 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.012477    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.013131 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.012518    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c275e-bd60-49f4-aa5a-1acca052b423-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.013131 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.012547    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.013131 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.012594    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"]
Oct 30 02:52:27.113654 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.113618    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.113825 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.113708    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.113825 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.113739    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.113825 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.113758    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.113825 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.113776    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ddg\" (UniqueName: \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-kube-api-access-m4ddg\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.113825 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.113804    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.114103 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.113877    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c275e-bd60-49f4-aa5a-1acca052b423-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.114103 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.113912    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.114103 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.113943    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd23c84-3409-4a96-b587-5d8270a17205-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.114103 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.113986    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.114103 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.114011    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.114103 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.114045    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.114386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.114144    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.114386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.114196    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.114386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.114263    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.114386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.114325    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.114386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.114367    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.114636 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.114400    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd23c84-3409-4a96-b587-5d8270a17205-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.114636 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.114437    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmdn\" (UniqueName: \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-kube-api-access-vvmdn\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.114636 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.114579    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c275e-bd60-49f4-aa5a-1acca052b423-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.114862 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.114830    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.115145 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.115126    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.115280 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.115214    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.116468 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.116450    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.116634 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.116619    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.133870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.133837    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.134825 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.134803    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmdn\" (UniqueName: \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-kube-api-access-vvmdn\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hc2qp\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.215106 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215039    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215211 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215119    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd23c84-3409-4a96-b587-5d8270a17205-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215211 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215152    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215329 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215204    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215329 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215233    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ddg\" (UniqueName: \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-kube-api-access-m4ddg\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215329 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215310    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215472 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215365    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd23c84-3409-4a96-b587-5d8270a17205-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215472 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215415    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215472 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215447    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215615 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215587    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215615 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215606    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215835 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215815    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.215913 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.215890    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.216134 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.216116    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd23c84-3409-4a96-b587-5d8270a17205-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.217580 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.217556    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.217734 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.217715    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd23c84-3409-4a96-b587-5d8270a17205-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.237187 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.237157    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ddg\" (UniqueName: \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-kube-api-access-m4ddg\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.237326 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.237310    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6qcgn\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.267589 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.267567    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:27.292761 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.292740    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:27.601071 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.601042    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"]
Oct 30 02:52:27.604544 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:52:27.604513    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2c275e_bd60_49f4_aa5a_1acca052b423.slice/crio-a51c83efa0b67a04abd0be3ca4f00b8e11fe3a6713d38a7badf9611c0e8614e4 WatchSource:0}: Error finding container a51c83efa0b67a04abd0be3ca4f00b8e11fe3a6713d38a7badf9611c0e8614e4: Status 404 returned error can't find the container with id a51c83efa0b67a04abd0be3ca4f00b8e11fe3a6713d38a7badf9611c0e8614e4
Oct 30 02:52:27.621483 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.621453    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"]
Oct 30 02:52:27.626723 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:52:27.626702    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd23c84_3409_4a96_b587_5d8270a17205.slice/crio-90889e7bff0242e15dec3fede20fa93686715dee6c7e91249648e019ee28bfd6 WatchSource:0}: Error finding container 90889e7bff0242e15dec3fede20fa93686715dee6c7e91249648e019ee28bfd6: Status 404 returned error can't find the container with id 90889e7bff0242e15dec3fede20fa93686715dee6c7e91249648e019ee28bfd6
Oct 30 02:52:27.830606 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.830567    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" event={"ID":"2bd23c84-3409-4a96-b587-5d8270a17205","Type":"ContainerStarted","Data":"90889e7bff0242e15dec3fede20fa93686715dee6c7e91249648e019ee28bfd6"}
Oct 30 02:52:27.831751 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:27.831727    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" event={"ID":"bb2c275e-bd60-49f4-aa5a-1acca052b423","Type":"ContainerStarted","Data":"a51c83efa0b67a04abd0be3ca4f00b8e11fe3a6713d38a7badf9611c0e8614e4"}
Oct 30 02:52:30.440225 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:30.440195    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:52:30.440487 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:30.440265    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:52:30.440487 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:30.440293    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:52:30.537835 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:30.537801    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:52:30.537941 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:30.537901    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:52:30.537983 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:30.537947    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:52:30.801252 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:30.801223    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-v8fgz"
Oct 30 02:52:30.849928 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:30.849891    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" event={"ID":"2bd23c84-3409-4a96-b587-5d8270a17205","Type":"ContainerStarted","Data":"5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419"}
Oct 30 02:52:30.852076 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:30.852045    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" event={"ID":"bb2c275e-bd60-49f4-aa5a-1acca052b423","Type":"ContainerStarted","Data":"ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b"}
Oct 30 02:52:30.903141 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:30.903074    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" podStartSLOduration=2.092254931 podStartE2EDuration="4.903053916s" totalImagesPullingTime="2.810798985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:30.901424779 +0000 UTC m=+696.203603137" watchObservedRunningTime="2026-10-30 02:52:30.903053916 +0000 UTC m=+696.205232282"
Oct 30 02:52:31.268929 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:31.268258    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:31.270555 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:31.270527    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 02:52:31.270679 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:31.270578    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 02:52:31.293396 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:31.293371    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:31.294505 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:31.294466    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:52:31.294590 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:31.294526    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 02:52:32.268994 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:32.268954    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 02:52:32.269503 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:32.269016    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 02:52:32.293926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:32.293901    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:52:32.294060 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:32.293970    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 02:52:32.659367 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:32.659327    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:32.781842 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:32.781785    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:33.269907 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:33.269051    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 02:52:33.269907 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:33.269277    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 02:52:33.294062 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:33.293970    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:52:33.294460 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:33.294132    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 02:52:34.268737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:34.268235    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 02:52:34.268737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:34.268309    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 02:52:34.293368 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:34.293341    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:52:34.293686 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:34.293382    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 02:52:35.268330 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:35.268289    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 02:52:35.268643 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:35.268612    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 02:52:35.293827 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:35.293801    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:52:35.294155 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:35.293871    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 02:52:36.268322 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:36.268281    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 02:52:36.268660 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:36.268621    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 02:52:36.293638 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:36.293615    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:52:36.293796 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:36.293780    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 02:52:37.268825 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:37.268785    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 02:52:37.269265 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:37.268843    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 02:52:37.293978 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:37.293936    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 02:52:37.294100 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:37.294004    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 02:52:38.857172 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:38.857134    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:38.857994 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:38.857968    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:38.858781 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:38.858757    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 02:52:38.921560 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:38.921508    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" podStartSLOduration=9.990961542 podStartE2EDuration="12.921495551s" totalImagesPullingTime="2.930534009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:30.952923145 +0000 UTC m=+696.255101509" watchObservedRunningTime="2026-10-30 02:52:38.921495551 +0000 UTC m=+704.223673912"
Oct 30 02:52:38.932491 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:38.932463    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:38.932810 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:38.932782    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:38.933505 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:38.933488    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:52:42.041691 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:42.041655    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:42.545588 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:42.545551    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:44.284797 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:44.284763    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:44.441044 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:44.441013    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:45.458207 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:45.458172    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:45.925666 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:45.925625    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:46.267782 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:52:46.267752    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:55.444804 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:52:55.444769    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:53:12.657136 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:12.657092    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:12.782077 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:12.782035    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:13.091036 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.091004    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"]
Oct 30 02:53:13.094627 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.094606    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.112119 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.112095    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-32bad3-openshift-default-dockercfg-wg2rd\""
Oct 30 02:53:13.124947 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.124922    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"]
Oct 30 02:53:13.134241 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.134213    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.134364 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.134242    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.134364 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.134272    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f654df01-c7f4-472a-a71b-6a43df523f3a-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.134492 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.134461    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.134545 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.134505    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.134596 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.134577    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.134631 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.134607    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdk2c\" (UniqueName: \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-kube-api-access-pdk2c\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.134729 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.134710    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.134795 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.134777    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.190318 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.190290    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:13.235636 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.235600    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.235767 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.235649    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.235767 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.235673    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.235767 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.235699    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdk2c\" (UniqueName: \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-kube-api-access-pdk2c\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.235767 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.235756    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.235984 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.235786    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.235984 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.235838    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.235984 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.235880    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.235984 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.235907    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f654df01-c7f4-472a-a71b-6a43df523f3a-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.236671 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.236641    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f654df01-c7f4-472a-a71b-6a43df523f3a-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.236951 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.236930    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.237890 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.237868    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.238354 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.238291    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.238573 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.238554    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.239408 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.239388    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.240041 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.240017    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.255119 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.255093    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.256686 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.256658    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdk2c\" (UniqueName: \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-kube-api-access-pdk2c\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.350593 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.350491    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:13.390580 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.390541    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"]
Oct 30 02:53:13.395866 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.395830    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.408110 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.408086    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:13.418911 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.418887    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"]
Oct 30 02:53:13.437207 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.437182    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.437325 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.437223    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.437325 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.437247    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.437447 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.437345    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.437447 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.437370    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5zbt\" (UniqueName: \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-kube-api-access-d5zbt\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.437447 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.437391    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.437601 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.437500    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.437601 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.437517    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.437601 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.437539    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.446968 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.446944    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f7d1a1-openshift-default-dockercfg-54qbh\""
Oct 30 02:53:13.529659 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.529629    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:13.538944 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.538911    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539082 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.538978    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539082 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539004    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539082 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539037    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539238 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539091    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539238 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539137    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539238 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539188    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539379 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539274    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539379 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539310    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5zbt\" (UniqueName: \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-kube-api-access-d5zbt\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539517 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539493    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539576 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539520    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539702 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539685    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539811 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539791    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.539905 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.539883    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.541456 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.541436    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.541543 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.541475    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.609620 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.609556    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.616169 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.616149    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5zbt\" (UniqueName: \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-kube-api-access-d5zbt\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.710372 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.710342    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:13.730740 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.730715    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"]
Oct 30 02:53:13.735748 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:53:13.735715    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf654df01_c7f4_472a_a71b_6a43df523f3a.slice/crio-94f573695d28c6ea800272c6029110ae23458d6620ee19df09d6a6b88f68bc5d WatchSource:0}: Error finding container 94f573695d28c6ea800272c6029110ae23458d6620ee19df09d6a6b88f68bc5d: Status 404 returned error can't find the container with id 94f573695d28c6ea800272c6029110ae23458d6620ee19df09d6a6b88f68bc5d
Oct 30 02:53:13.738359 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.738325    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:13.738467 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.738429    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:13.738531 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.738490    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:13.767566 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.767495    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"]
Oct 30 02:53:13.774880 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.774392    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.782367 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.782329    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:13.800080 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.796386    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e11f03-openshift-default-dockercfg-ph97z\""
Oct 30 02:53:13.800080 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.799758    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"]
Oct 30 02:53:13.845238 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.845202    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.845414 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.845272    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.845414 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.845395    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.845513 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.845447    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.845548 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.845536    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4tm\" (UniqueName: \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-kube-api-access-4g4tm\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.845635 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.845613    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.845672 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.845656    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.845705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.845683    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8387b9-7f57-4719-935c-1515c8b21ab4-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.845735 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.845720    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.938832 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.938707    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:13.947320 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.947232    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.947320 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.947281    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.947541 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.947405    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.947541 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.947474    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.947541 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.947514    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4tm\" (UniqueName: \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-kube-api-access-4g4tm\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.947710 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.947599    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.947710 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.947647    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.947710 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.947673    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8387b9-7f57-4719-935c-1515c8b21ab4-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.947710 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.947698    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.947936 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.947708    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.948405 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.948365    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.948715 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.948690    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.948795 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.948773    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.949142 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.949114    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8387b9-7f57-4719-935c-1515c8b21ab4-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.950520 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.950499    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.951312 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.951288    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.968757 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.968721    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:13.969146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:13.969120    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4tm\" (UniqueName: \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-kube-api-access-4g4tm\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:14.033483 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.033449    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"]
Oct 30 02:53:14.038378 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:53:14.038333    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7c2d96_e8aa_4ad8_8d93_8d3dbaa82528.slice/crio-94ea690f253f3eea00f4c6703ebfdbb65110d6b13e9dfe79b16b2ca07ebd6ddf WatchSource:0}: Error finding container 94ea690f253f3eea00f4c6703ebfdbb65110d6b13e9dfe79b16b2ca07ebd6ddf: Status 404 returned error can't find the container with id 94ea690f253f3eea00f4c6703ebfdbb65110d6b13e9dfe79b16b2ca07ebd6ddf
Oct 30 02:53:14.040911 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.040878    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:14.041014 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.040942    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:14.041014 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.040971    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:14.063032 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.063004    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" event={"ID":"f654df01-c7f4-472a-a71b-6a43df523f3a","Type":"ContainerStarted","Data":"bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed"}
Oct 30 02:53:14.063151 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.063060    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" event={"ID":"f654df01-c7f4-472a-a71b-6a43df523f3a","Type":"ContainerStarted","Data":"94f573695d28c6ea800272c6029110ae23458d6620ee19df09d6a6b88f68bc5d"}
Oct 30 02:53:14.064335 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.064309    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" event={"ID":"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528","Type":"ContainerStarted","Data":"94ea690f253f3eea00f4c6703ebfdbb65110d6b13e9dfe79b16b2ca07ebd6ddf"}
Oct 30 02:53:14.091048 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.091024    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"]
Oct 30 02:53:14.096070 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.096051    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.108745 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.108723    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:14.115171 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.115122    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" podStartSLOduration=1.11510504 podStartE2EDuration="1.11510504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:14.113397586 +0000 UTC m=+739.415575956" watchObservedRunningTime="2026-10-30 02:53:14.11510504 +0000 UTC m=+739.417283409"
Oct 30 02:53:14.116642 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.116620    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-810119-openshift-default-dockercfg-84mrx\""
Oct 30 02:53:14.150772 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.150742    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"]
Oct 30 02:53:14.157232 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.157196    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.157389 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.157244    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.157389 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.157365    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.157508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.157467    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htvdj\" (UniqueName: \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-kube-api-access-htvdj\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.159552 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.159466    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.159650 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.159601    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.159691 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.159675    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.159829 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.159781    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.159961 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.159920    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.195562 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.194518    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.261423    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htvdj\" (UniqueName: \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-kube-api-access-htvdj\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.261506    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.261539    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.261568    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.261616    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.261659    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.261697    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.261732    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.261817    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.262158    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.262231    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.262511    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.263074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.262758    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.265157 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.265116    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.270070 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.269983    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.288311 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.288287    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.295839 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.295815    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.304585 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.304559    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htvdj\" (UniqueName: \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-kube-api-access-htvdj\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.404980 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.404946    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:14.408853 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.408829    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:14.413487 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.413466    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:14.413618 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.413506    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:14.428692 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.428658    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"]
Oct 30 02:53:14.432430 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:53:14.432392    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8387b9_7f57_4719_935c_1515c8b21ab4.slice/crio-7904fb97e0a27664ecfe674debc9855a3ddedef539b741521a0826305056a09a WatchSource:0}: Error finding container 7904fb97e0a27664ecfe674debc9855a3ddedef539b741521a0826305056a09a: Status 404 returned error can't find the container with id 7904fb97e0a27664ecfe674debc9855a3ddedef539b741521a0826305056a09a
Oct 30 02:53:14.435245 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.435218    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:14.435309 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.435278    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:14.435309 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.435305    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:14.541052 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.541020    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:14.753283 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.753190    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"]
Oct 30 02:53:14.758121 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:53:14.758090    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f1f00e_6654_4a46_aa16_6e3a38508d6c.slice/crio-cb6b8b4a12685103dea32246c5b668f898ef2e0b78a12a667d4634d4fe3af9fc WatchSource:0}: Error finding container cb6b8b4a12685103dea32246c5b668f898ef2e0b78a12a667d4634d4fe3af9fc: Status 404 returned error can't find the container with id cb6b8b4a12685103dea32246c5b668f898ef2e0b78a12a667d4634d4fe3af9fc
Oct 30 02:53:14.760431 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.760400    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:14.760504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.760462    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:14.760561 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:14.760506    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:15.071704 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.071671    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" event={"ID":"8b8387b9-7f57-4719-935c-1515c8b21ab4","Type":"ContainerStarted","Data":"428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22"}
Oct 30 02:53:15.071934 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.071729    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" event={"ID":"8b8387b9-7f57-4719-935c-1515c8b21ab4","Type":"ContainerStarted","Data":"7904fb97e0a27664ecfe674debc9855a3ddedef539b741521a0826305056a09a"}
Oct 30 02:53:15.073325 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.073297    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" event={"ID":"87f1f00e-6654-4a46-aa16-6e3a38508d6c","Type":"ContainerStarted","Data":"b87e8ba7fa42a0a357f1755c211ebd0b0ffc8d20d5ad5a1e6e547d361532a461"}
Oct 30 02:53:15.073448 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.073345    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" event={"ID":"87f1f00e-6654-4a46-aa16-6e3a38508d6c","Type":"ContainerStarted","Data":"cb6b8b4a12685103dea32246c5b668f898ef2e0b78a12a667d4634d4fe3af9fc"}
Oct 30 02:53:15.074819 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.074796    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" event={"ID":"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528","Type":"ContainerStarted","Data":"5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a"}
Oct 30 02:53:15.075071 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.075052    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:15.076010 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.075994    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:15.109686 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.109663    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:15.113692 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.113655    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" podStartSLOduration=2.113642103 podStartE2EDuration="2.113642103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:15.111875486 +0000 UTC m=+740.414053842" watchObservedRunningTime="2026-10-30 02:53:15.113642103 +0000 UTC m=+740.415820467"
Oct 30 02:53:15.115644 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.115627    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:15.189277 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.189063    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" podStartSLOduration=2.189049281 podStartE2EDuration="2.189049281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:15.187654494 +0000 UTC m=+740.489832859" watchObservedRunningTime="2026-10-30 02:53:15.189049281 +0000 UTC m=+740.491227646"
Oct 30 02:53:15.228054 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.228001    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" podStartSLOduration=1.22798782 podStartE2EDuration="1.22798782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:15.226568082 +0000 UTC m=+740.528746449" watchObservedRunningTime="2026-10-30 02:53:15.22798782 +0000 UTC m=+740.530166182"
Oct 30 02:53:15.413939 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.413858    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:15.418801 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.418783    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:15.710767 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.710678    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:15.715791 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:15.715768    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:16.080452 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:16.080419    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:16.080916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:16.080490    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:16.080916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:16.080519    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:16.080916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:16.080719    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:16.080916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:16.080766    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:16.080916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:16.080799    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:16.866751 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:16.866689    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:17.049470 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:17.049439    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:17.333893 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:17.333842    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:17.867197 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:17.867145    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:18.130625 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.130504    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:18.302316 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.302278    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:18.558674 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.558644    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"]
Oct 30 02:53:18.562913 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.562891    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.603450 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.603421    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njv5j\" (UniqueName: \"kubernetes.io/projected/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-kube-api-access-njv5j\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.603559 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.603477    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-metrics-certs\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.603609 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.603575    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.603652 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.603617    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-maas-api-tls\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.607331 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.607307    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"]
Oct 30 02:53:18.630955 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.630922    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-32bad3-serving-cert\""
Oct 30 02:53:18.704917 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.704877    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njv5j\" (UniqueName: \"kubernetes.io/projected/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-kube-api-access-njv5j\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.705080 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.704965    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-metrics-certs\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.705080 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.705023    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.705215 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.705181    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-maas-api-tls\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.705584 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:53:18.705301    5061 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-serving-cert: secret "maas-api-e2e-worker-w6-32bad3-serving-cert" not found
Oct 30 02:53:18.705584 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:53:18.705370    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-maas-api-tls podName:39ff9afe-ec5c-4de9-8e0b-be98f8ccc005 nodeName:}" failed. No retries permitted until 2026-10-30 02:53:19.205349966 +0000 UTC m=+744.507528331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-maas-api-tls") pod "maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7" (UID: "39ff9afe-ec5c-4de9-8e0b-be98f8ccc005") : secret "maas-api-e2e-worker-w6-32bad3-serving-cert" not found
Oct 30 02:53:18.706593 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.706571    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.714451 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.714428    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-metrics-certs\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.740301 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.740274    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njv5j\" (UniqueName: \"kubernetes.io/projected/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-kube-api-access-njv5j\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:18.952305 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:18.952214    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:19.132902 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:19.132868    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:19.210917 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:19.210820    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-maas-api-tls\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:19.213206 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:19.213180    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-maas-api-tls\") pod \"maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:19.475972 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:19.475893    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:19.820993 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:19.820897    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"]
Oct 30 02:53:19.866596 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:19.866569    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:20.101239 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:20.101161    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7" event={"ID":"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005","Type":"ContainerStarted","Data":"ca40b1a7b893bfd60c181d2704800cf451f925591d58ac6a6ae5dac91e5f00ea"}
Oct 30 02:53:20.202168 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:20.202105    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:20.481881 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:20.481767    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:20.894773 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:20.894741    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:21.107523 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.107493    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7" event={"ID":"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005","Type":"ContainerStarted","Data":"c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d"}
Oct 30 02:53:21.107699 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.107621    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:21.169128 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.169032    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7" podStartSLOduration=2.806589069 podStartE2EDuration="3.169019095s" totalImagesPullingTime="362.430026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:21.167962699 +0000 UTC m=+746.470141064" watchObservedRunningTime="2026-10-30 02:53:21.169019095 +0000 UTC m=+746.471197457"
Oct 30 02:53:21.527321 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.527249    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:21.700381 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.700349    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"]
Oct 30 02:53:21.703918 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.703896    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 02:53:21.749872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.737330    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d9f3ac\""
Oct 30 02:53:21.749872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.737479    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-d9f3ac-dockercfg-gfgcv\""
Oct 30 02:53:21.769895 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.767227    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"]
Oct 30 02:53:21.837317 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.837286    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x77sq\" (UniqueName: \"kubernetes.io/projected/6d01fd3e-ed78-4ad5-b081-89effbaf2338-kube-api-access-x77sq\") pod \"payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn\" (UID: \"6d01fd3e-ed78-4ad5-b081-89effbaf2338\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 02:53:21.837513 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.837356    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d01fd3e-ed78-4ad5-b081-89effbaf2338-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn\" (UID: \"6d01fd3e-ed78-4ad5-b081-89effbaf2338\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 02:53:21.858036 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.858007    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:21.939044 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.939005    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d01fd3e-ed78-4ad5-b081-89effbaf2338-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn\" (UID: \"6d01fd3e-ed78-4ad5-b081-89effbaf2338\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 02:53:21.939430 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.939126    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x77sq\" (UniqueName: \"kubernetes.io/projected/6d01fd3e-ed78-4ad5-b081-89effbaf2338-kube-api-access-x77sq\") pod \"payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn\" (UID: \"6d01fd3e-ed78-4ad5-b081-89effbaf2338\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 02:53:21.939703 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.939685    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d01fd3e-ed78-4ad5-b081-89effbaf2338-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn\" (UID: \"6d01fd3e-ed78-4ad5-b081-89effbaf2338\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 02:53:21.966945 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:21.966919    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x77sq\" (UniqueName: \"kubernetes.io/projected/6d01fd3e-ed78-4ad5-b081-89effbaf2338-kube-api-access-x77sq\") pod \"payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn\" (UID: \"6d01fd3e-ed78-4ad5-b081-89effbaf2338\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 02:53:22.019828 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:22.019804    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 02:53:22.099462 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:22.099384    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:22.317019 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:22.316987    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"]
Oct 30 02:53:22.320473 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:53:22.320444    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d01fd3e_ed78_4ad5_b081_89effbaf2338.slice/crio-d6276b8aba1a98a6e1bcb6fc674f8454268c8aa553eeeed151b8d5fd010f462f WatchSource:0}: Error finding container d6276b8aba1a98a6e1bcb6fc674f8454268c8aa553eeeed151b8d5fd010f462f: Status 404 returned error can't find the container with id d6276b8aba1a98a6e1bcb6fc674f8454268c8aa553eeeed151b8d5fd010f462f
Oct 30 02:53:23.119379 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:23.119336    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn" event={"ID":"6d01fd3e-ed78-4ad5-b081-89effbaf2338","Type":"ContainerStarted","Data":"7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba"}
Oct 30 02:53:23.119800 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:23.119398    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn" event={"ID":"6d01fd3e-ed78-4ad5-b081-89effbaf2338","Type":"ContainerStarted","Data":"d6276b8aba1a98a6e1bcb6fc674f8454268c8aa553eeeed151b8d5fd010f462f"}
Oct 30 02:53:23.119800 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:23.119505    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 02:53:23.157352 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:23.157298    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn" podStartSLOduration=2.157284489 podStartE2EDuration="2.157284489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:23.15595436 +0000 UTC m=+748.458132721" watchObservedRunningTime="2026-10-30 02:53:23.157284489 +0000 UTC m=+748.459462865"
Oct 30 02:53:23.563821 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:23.563733    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:23.806183 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:23.806129    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:24.053699 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.053652    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"]
Oct 30 02:53:24.058123 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.058096    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.074947 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.074875    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-aeff1d-serving-cert\""
Oct 30 02:53:24.083981 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.083951    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"]
Oct 30 02:53:24.165654 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.165610    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsq88\" (UniqueName: \"kubernetes.io/projected/d1a22ae9-6252-4363-b95b-fa3708998135-kube-api-access-gsq88\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.166037 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.165658    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-maas-api-tls\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.166037 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.165869    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-metrics-certs\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.166037 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.165944    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a22ae9-6252-4363-b95b-fa3708998135-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.189785 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.189756    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:24.267140 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.267105    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-metrics-certs\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.267355 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.267155    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a22ae9-6252-4363-b95b-fa3708998135-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.267355 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.267200    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsq88\" (UniqueName: \"kubernetes.io/projected/d1a22ae9-6252-4363-b95b-fa3708998135-kube-api-access-gsq88\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.267355 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.267225    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-maas-api-tls\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.267961 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.267936    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a22ae9-6252-4363-b95b-fa3708998135-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.269695 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.269658    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-maas-api-tls\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.269786 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.269715    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-metrics-certs\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.285493 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.285466    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsq88\" (UniqueName: \"kubernetes.io/projected/d1a22ae9-6252-4363-b95b-fa3708998135-kube-api-access-gsq88\") pod \"maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.370187 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.370092    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:24.374139 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.374122    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:24.751466 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:24.751368    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"]
Oct 30 02:53:24.755700 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:53:24.755669    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a22ae9_6252_4363_b95b_fa3708998135.slice/crio-7914e116278fb405e8a1f4c9bf0a38c4aee8d77529cf2687f6fb3e62c413e9bf WatchSource:0}: Error finding container 7914e116278fb405e8a1f4c9bf0a38c4aee8d77529cf2687f6fb3e62c413e9bf: Status 404 returned error can't find the container with id 7914e116278fb405e8a1f4c9bf0a38c4aee8d77529cf2687f6fb3e62c413e9bf
Oct 30 02:53:25.150315 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.150287    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq" event={"ID":"d1a22ae9-6252-4363-b95b-fa3708998135","Type":"ContainerStarted","Data":"7914e116278fb405e8a1f4c9bf0a38c4aee8d77529cf2687f6fb3e62c413e9bf"}
Oct 30 02:53:25.463063 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.462912    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:25.475054 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:53:25.473614    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:53:25.593693 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.593657    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:25.715356 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.715260    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"]
Oct 30 02:53:25.727160 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.727133    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 02:53:25.744950 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.744709    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"]
Oct 30 02:53:25.753938 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.752080    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-810119\""
Oct 30 02:53:25.759624 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.759603    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-810119-dockercfg-bbzgw\""
Oct 30 02:53:25.781359 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.781336    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7jqf\" (UniqueName: \"kubernetes.io/projected/e44c0f23-392b-4bee-8c03-82f0b23abe90-kube-api-access-m7jqf\") pod \"payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs\" (UID: \"e44c0f23-392b-4bee-8c03-82f0b23abe90\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 02:53:25.781504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.781485    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44c0f23-392b-4bee-8c03-82f0b23abe90-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs\" (UID: \"e44c0f23-392b-4bee-8c03-82f0b23abe90\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 02:53:25.882109 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.882070    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7jqf\" (UniqueName: \"kubernetes.io/projected/e44c0f23-392b-4bee-8c03-82f0b23abe90-kube-api-access-m7jqf\") pod \"payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs\" (UID: \"e44c0f23-392b-4bee-8c03-82f0b23abe90\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 02:53:25.882276 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.882183    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44c0f23-392b-4bee-8c03-82f0b23abe90-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs\" (UID: \"e44c0f23-392b-4bee-8c03-82f0b23abe90\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 02:53:25.882900 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.882881    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44c0f23-392b-4bee-8c03-82f0b23abe90-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs\" (UID: \"e44c0f23-392b-4bee-8c03-82f0b23abe90\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 02:53:25.906526 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:25.906493    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7jqf\" (UniqueName: \"kubernetes.io/projected/e44c0f23-392b-4bee-8c03-82f0b23abe90-kube-api-access-m7jqf\") pod \"payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs\" (UID: \"e44c0f23-392b-4bee-8c03-82f0b23abe90\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 02:53:26.063360 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.063323    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 02:53:26.159233 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.159169    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq" event={"ID":"d1a22ae9-6252-4363-b95b-fa3708998135","Type":"ContainerStarted","Data":"8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e"}
Oct 30 02:53:26.159410 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.159271    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:26.212864 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.212795    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"]
Oct 30 02:53:26.217627 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.217604    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.248373 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.248313    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq" podStartSLOduration=1.898606751 podStartE2EDuration="2.248292219s" totalImagesPullingTime="349.685468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:26.209521337 +0000 UTC m=+751.511699715" watchObservedRunningTime="2026-10-30 02:53:26.248292219 +0000 UTC m=+751.550470585"
Oct 30 02:53:26.262979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.262948    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"]
Oct 30 02:53:26.270968 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.270941    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-7d1788-serving-cert\""
Oct 30 02:53:26.285565 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.285530    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9rs4\" (UniqueName: \"kubernetes.io/projected/2c362050-6a7f-4339-b028-8ac00f20cc9e-kube-api-access-c9rs4\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.285730 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.285647    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-metrics-certs\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.285829 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.285797    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c362050-6a7f-4339-b028-8ac00f20cc9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.285956 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.285940    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.373651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.373575    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"]
Oct 30 02:53:26.378430 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:53:26.378404    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44c0f23_392b_4bee_8c03_82f0b23abe90.slice/crio-37aa62d4aeaa4bfc59244f609b990dd6fdd19d42fb0f58adc4125b3b0a369be9 WatchSource:0}: Error finding container 37aa62d4aeaa4bfc59244f609b990dd6fdd19d42fb0f58adc4125b3b0a369be9: Status 404 returned error can't find the container with id 37aa62d4aeaa4bfc59244f609b990dd6fdd19d42fb0f58adc4125b3b0a369be9
Oct 30 02:53:26.386969 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.386941    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c362050-6a7f-4339-b028-8ac00f20cc9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.387056 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.386996    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.387095 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.387070    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9rs4\" (UniqueName: \"kubernetes.io/projected/2c362050-6a7f-4339-b028-8ac00f20cc9e-kube-api-access-c9rs4\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.387179 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.387164    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-metrics-certs\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.387730 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.387706    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c362050-6a7f-4339-b028-8ac00f20cc9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.389596 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.389568    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-metrics-certs\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.389667 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.389568    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.445584 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.445545    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9rs4\" (UniqueName: \"kubernetes.io/projected/2c362050-6a7f-4339-b028-8ac00f20cc9e-kube-api-access-c9rs4\") pod \"maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.461296 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.460277    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:26.563500 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.563459    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:26.656560 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.656534    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:26.874574 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.874517    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:26.913381 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:26.913300    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"]
Oct 30 02:53:26.918646 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:53:26.918617    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c362050_6a7f_4339_b028_8ac00f20cc9e.slice/crio-89d727590e9cb47fad106a8a33ab83a1bea21a4b89d023264e5f6ff03fff3082 WatchSource:0}: Error finding container 89d727590e9cb47fad106a8a33ab83a1bea21a4b89d023264e5f6ff03fff3082: Status 404 returned error can't find the container with id 89d727590e9cb47fad106a8a33ab83a1bea21a4b89d023264e5f6ff03fff3082
Oct 30 02:53:27.118627 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:27.118601    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 02:53:27.166581 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:27.166494    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs" event={"ID":"e44c0f23-392b-4bee-8c03-82f0b23abe90","Type":"ContainerStarted","Data":"56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6"}
Oct 30 02:53:27.166581 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:27.166544    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs" event={"ID":"e44c0f23-392b-4bee-8c03-82f0b23abe90","Type":"ContainerStarted","Data":"37aa62d4aeaa4bfc59244f609b990dd6fdd19d42fb0f58adc4125b3b0a369be9"}
Oct 30 02:53:27.166581 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:27.166576    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 02:53:27.167966 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:27.167945    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps" event={"ID":"2c362050-6a7f-4339-b028-8ac00f20cc9e","Type":"ContainerStarted","Data":"89d727590e9cb47fad106a8a33ab83a1bea21a4b89d023264e5f6ff03fff3082"}
Oct 30 02:53:27.206374 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:27.206323    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs" podStartSLOduration=2.2063092380000002 podStartE2EDuration="2.206309238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:27.204247741 +0000 UTC m=+752.506426107" watchObservedRunningTime="2026-10-30 02:53:27.206309238 +0000 UTC m=+752.508487603"
Oct 30 02:53:28.143870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.143322    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"]
Oct 30 02:53:28.148442 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.148419    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.158356 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.158329    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"]
Oct 30 02:53:28.166272 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.166247    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-d9f3ac-serving-cert\""
Oct 30 02:53:28.177555 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.177534    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps" event={"ID":"2c362050-6a7f-4339-b028-8ac00f20cc9e","Type":"ContainerStarted","Data":"c85d16c4f35915ffebb5edf529b7470448037289e393ae30855e838a9086dd7e"}
Oct 30 02:53:28.177683 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.177578    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:28.207361 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.207327    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-metrics-certs\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.207509 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.207368    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.207509 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.207458    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbsp\" (UniqueName: \"kubernetes.io/projected/8aa34fac-44e6-44c0-9666-f0b9d5974a76-kube-api-access-xhbsp\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.207777 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.207567    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa34fac-44e6-44c0-9666-f0b9d5974a76-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.251984 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.251920    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps" podStartSLOduration=1.908542077 podStartE2EDuration="2.251898312s" totalImagesPullingTime="343.356235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:28.244213649 +0000 UTC m=+753.546392018" watchObservedRunningTime="2026-10-30 02:53:28.251898312 +0000 UTC m=+753.554076677"
Oct 30 02:53:28.289146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.289094    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:28.308456 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.308414    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-metrics-certs\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.308718 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.308688    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.308843 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.308755    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbsp\" (UniqueName: \"kubernetes.io/projected/8aa34fac-44e6-44c0-9666-f0b9d5974a76-kube-api-access-xhbsp\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.308843 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.308787    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa34fac-44e6-44c0-9666-f0b9d5974a76-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.308843 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:53:28.308811    5061 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-serving-cert: secret "maas-api-e2e-worker-w3-d9f3ac-serving-cert" not found
Oct 30 02:53:28.309044 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:53:28.308892    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-maas-api-tls podName:8aa34fac-44e6-44c0-9666-f0b9d5974a76 nodeName:}" failed. No retries permitted until 2026-10-30 02:53:28.808870836 +0000 UTC m=+754.111049181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-maas-api-tls") pod "maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45" (UID: "8aa34fac-44e6-44c0-9666-f0b9d5974a76") : secret "maas-api-e2e-worker-w3-d9f3ac-serving-cert" not found
Oct 30 02:53:28.309443 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.309420    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa34fac-44e6-44c0-9666-f0b9d5974a76-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.311064 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.311039    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-metrics-certs\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.354461 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.354426    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbsp\" (UniqueName: \"kubernetes.io/projected/8aa34fac-44e6-44c0-9666-f0b9d5974a76-kube-api-access-xhbsp\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.448614 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.448478    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:28.679887 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.679825    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:28.855376 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.855343    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:28.857735 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:28.857710    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:29.064284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:29.064244    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:29.387404 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:29.387311    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"]
Oct 30 02:53:29.391935 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:53:29.391905    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa34fac_44e6_44c0_9666_f0b9d5974a76.slice/crio-40ea5af3f04dcbe2ae5bab40bb9b8d4d7d5ee609bee74086f050f2d4b5f7c449 WatchSource:0}: Error finding container 40ea5af3f04dcbe2ae5bab40bb9b8d4d7d5ee609bee74086f050f2d4b5f7c449: Status 404 returned error can't find the container with id 40ea5af3f04dcbe2ae5bab40bb9b8d4d7d5ee609bee74086f050f2d4b5f7c449
Oct 30 02:53:29.641524 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:29.641453    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:29.833258 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:29.833210    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:30.188752 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:30.188717    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45" event={"ID":"8aa34fac-44e6-44c0-9666-f0b9d5974a76","Type":"ContainerStarted","Data":"3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b"}
Oct 30 02:53:30.188944 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:30.188772    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45" event={"ID":"8aa34fac-44e6-44c0-9666-f0b9d5974a76","Type":"ContainerStarted","Data":"40ea5af3f04dcbe2ae5bab40bb9b8d4d7d5ee609bee74086f050f2d4b5f7c449"}
Oct 30 02:53:30.188944 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:30.188907    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:30.225731 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:30.225681    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45" podStartSLOduration=1.909634257 podStartE2EDuration="2.225667783s" totalImagesPullingTime="316.033526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:30.225092688 +0000 UTC m=+755.527271054" watchObservedRunningTime="2026-10-30 02:53:30.225667783 +0000 UTC m=+755.527846146"
Oct 30 02:53:30.413717 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:30.413681    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:31.117017 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:31.116979    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:53:31.117203 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:31.117043    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:53:32.173834 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.173802    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 02:53:32.392242 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.392210    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"]
Oct 30 02:53:32.396570 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.396548    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.404657 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.404633    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.404771 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.404687    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c86dfa-68e3-479a-894d-8655a536eda0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.404771 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.404708    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-metrics-certs\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.404909 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.404895    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9tg\" (UniqueName: \"kubernetes.io/projected/b9c86dfa-68e3-479a-894d-8655a536eda0-kube-api-access-4t9tg\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.418233 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.418204    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-e11f03-serving-cert\""
Oct 30 02:53:32.429788 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.429715    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"]
Oct 30 02:53:32.505744 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.505710    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9tg\" (UniqueName: \"kubernetes.io/projected/b9c86dfa-68e3-479a-894d-8655a536eda0-kube-api-access-4t9tg\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.505944 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.505760    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.505944 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.505797    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c86dfa-68e3-479a-894d-8655a536eda0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.505944 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.505821    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-metrics-certs\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.505944 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:53:32.505924    5061 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-serving-cert: secret "maas-api-e2e-worker-w1-e11f03-serving-cert" not found
Oct 30 02:53:32.506199 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:53:32.506005    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-maas-api-tls podName:b9c86dfa-68e3-479a-894d-8655a536eda0 nodeName:}" failed. No retries permitted until 2026-10-30 02:53:33.005988709 +0000 UTC m=+758.308167054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-maas-api-tls") pod "maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6" (UID: "b9c86dfa-68e3-479a-894d-8655a536eda0") : secret "maas-api-e2e-worker-w1-e11f03-serving-cert" not found
Oct 30 02:53:32.506566 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.506543    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c86dfa-68e3-479a-894d-8655a536eda0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.508302 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.508282    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-metrics-certs\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.525108 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.525076    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9tg\" (UniqueName: \"kubernetes.io/projected/b9c86dfa-68e3-479a-894d-8655a536eda0-kube-api-access-4t9tg\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:32.529808 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.529782    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:32.661955 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:32.661918    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:33.010368 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:33.010328    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:33.013114 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:33.013088    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:33.206433 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:33.206332    5061 generic.go:356] "Generic (PLEG): container finished" podID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerID="bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed" exitCode=0
Oct 30 02:53:33.206433 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:33.206401    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" event={"ID":"f654df01-c7f4-472a-a71b-6a43df523f3a","Type":"ContainerDied","Data":"bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed"}
Oct 30 02:53:33.206993 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:33.206896    5061 scope.go:122] "RemoveContainer" containerID="bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed"
Oct 30 02:53:33.208243 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:33.208214    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:33.208358 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:33.208275    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:33.208358 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:33.208308    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:33.310617 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:33.310592    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:33.639017 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:33.638986    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"]
Oct 30 02:53:33.642924 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:53:33.642890    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c86dfa_68e3_479a_894d_8655a536eda0.slice/crio-ea273549a17babab332a392953d562b19628abbdb7333be746838344a63109a7 WatchSource:0}: Error finding container ea273549a17babab332a392953d562b19628abbdb7333be746838344a63109a7: Status 404 returned error can't find the container with id ea273549a17babab332a392953d562b19628abbdb7333be746838344a63109a7
Oct 30 02:53:34.147013 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:34.146981    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 02:53:34.189180 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:34.189029    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 02:53:34.215736 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:34.215653    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6" event={"ID":"b9c86dfa-68e3-479a-894d-8655a536eda0","Type":"ContainerStarted","Data":"a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818"}
Oct 30 02:53:34.216462 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:34.216440    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6" event={"ID":"b9c86dfa-68e3-479a-894d-8655a536eda0","Type":"ContainerStarted","Data":"ea273549a17babab332a392953d562b19628abbdb7333be746838344a63109a7"}
Oct 30 02:53:34.216561 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:34.216478    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:34.217651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:34.217629    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" event={"ID":"f654df01-c7f4-472a-a71b-6a43df523f3a","Type":"ContainerStarted","Data":"7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6"}
Oct 30 02:53:34.293417 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:34.293352    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6" podStartSLOduration=1.973032654 podStartE2EDuration="2.293336414s" totalImagesPullingTime="320.30376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:34.291582675 +0000 UTC m=+759.593761040" watchObservedRunningTime="2026-10-30 02:53:34.293336414 +0000 UTC m=+759.595514776"
Oct 30 02:53:34.409187 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:34.409141    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:34.409360 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:34.409315    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 02:53:34.409414 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:34.409364    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 02:53:35.410196 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:35.410081    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 02:53:35.410196 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:35.410146    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 02:53:35.472323 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:35.472286    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:35.664604 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:35.664521    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:36.200295 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:36.200268    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 02:53:36.408612 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:36.408577    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 02:53:36.408785 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:36.408628    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 02:53:36.709532 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:36.709500    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:37.397344 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:37.397308    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:37.408396 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:37.408369    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 02:53:37.408512 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:37.408432    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 02:53:37.559961 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:37.558654    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:37.699806 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:37.699720    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:37.858760 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:37.858725    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:38.180901 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:38.180872    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 02:53:38.409412 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:38.409377    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 02:53:38.409585 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:38.409428    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 02:53:39.459234 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:39.459185    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:39.460068 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:39.460041    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:39.461461 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:39.461435    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 02:53:39.861983 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:39.861935    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:40.191885 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:40.191723    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:40.261290 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:40.261263    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 02:53:41.675205 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:41.675170    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:43.552640 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:43.552605    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:45.649627 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:45.649527    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:46.111881 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:46.111837    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:48.469931 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:48.469889    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:48.599165 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:48.599120    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:49.146749 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:49.146699    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:49.357825 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:49.357789    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:50.745310 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:50.745239    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:51.210668 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.210632    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:51.377713 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.377684    5061 generic.go:356] "Generic (PLEG): container finished" podID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerID="428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22" exitCode=0
Oct 30 02:53:51.377934 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.377756    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" event={"ID":"8b8387b9-7f57-4719-935c-1515c8b21ab4","Type":"ContainerDied","Data":"428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22"}
Oct 30 02:53:51.378222 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.378202    5061 scope.go:122] "RemoveContainer" containerID="428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22"
Oct 30 02:53:51.379552 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.379531    5061 generic.go:356] "Generic (PLEG): container finished" podID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerID="5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a" exitCode=0
Oct 30 02:53:51.379654 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.379555    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:51.379654 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.379573    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" event={"ID":"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528","Type":"ContainerDied","Data":"5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a"}
Oct 30 02:53:51.379654 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.379613    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:51.379654 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.379642    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:51.379875 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.379861    5061 scope.go:122] "RemoveContainer" containerID="5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a"
Oct 30 02:53:51.380979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.380955    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:51.381041 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.381012    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:51.381086 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:51.381042    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:52.387134 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:52.386721    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" event={"ID":"8b8387b9-7f57-4719-935c-1515c8b21ab4","Type":"ContainerStarted","Data":"88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e"}
Oct 30 02:53:52.389146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:52.389122    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" event={"ID":"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528","Type":"ContainerStarted","Data":"4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2"}
Oct 30 02:53:52.390748 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:52.390725    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:52.502994 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:52.502958    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:52.711504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:52.711417    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:52.711692 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:52.711560    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 02:53:52.711692 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:52.711608    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 02:53:53.079235 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:53.079194    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:53.109320 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:53.109283    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:53.109491 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:53.109464    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:53:53.109544 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:53.109511    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 02:53:53.238872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:53.238180    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:53.535923 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:53.532383    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:53.711061 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:53.711032    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 02:53:53.711275 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:53.711252    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 02:53:53.931731 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:53.931693    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:54.110094 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:54.110061    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:53:54.110318 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:54.110294    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 02:53:54.156251 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:54.144527    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:54.276954 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:54.276821    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:54.542928 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:54.542262    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:54.710925 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:54.710893    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 02:53:54.711087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:54.710941    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 02:53:55.109143 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:55.109104    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:53:55.109328 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:55.109183    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 02:53:55.511448 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:53:55.511366    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:53:55.711031 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:55.710991    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 02:53:55.711520 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:55.711060    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 02:53:56.071815 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:56.071746    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:56.109603 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:56.109561    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:53:56.109797 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:56.109640    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 02:53:56.711344 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:56.711307    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 02:53:56.712012 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:56.711356    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 02:53:57.110038 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:57.110016    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 02:53:57.110174 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:57.110059    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 02:53:57.424789 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:57.424690    5061 generic.go:356] "Generic (PLEG): container finished" podID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerID="b87e8ba7fa42a0a357f1755c211ebd0b0ffc8d20d5ad5a1e6e547d361532a461" exitCode=0
Oct 30 02:53:57.424789 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:57.424735    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" event={"ID":"87f1f00e-6654-4a46-aa16-6e3a38508d6c","Type":"ContainerDied","Data":"b87e8ba7fa42a0a357f1755c211ebd0b0ffc8d20d5ad5a1e6e547d361532a461"}
Oct 30 02:53:57.425328 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:57.425305    5061 scope.go:122] "RemoveContainer" containerID="b87e8ba7fa42a0a357f1755c211ebd0b0ffc8d20d5ad5a1e6e547d361532a461"
Oct 30 02:53:57.426975 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:57.426951    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:57.427039 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:57.427020    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:57.427075 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:57.427053    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:53:58.000087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:58.000046    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:58.055961 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:58.055592    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:58.056669 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:58.056524    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:58.058456 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:58.058435    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 02:53:58.113332 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:58.113304    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:58.113599 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:58.113580    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:58.114224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:58.114206    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 02:53:58.303013 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:58.302984    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:58.446325 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:58.446287    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:58.448063 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:58.448029    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" event={"ID":"87f1f00e-6654-4a46-aa16-6e3a38508d6c","Type":"ContainerStarted","Data":"e762852e18318669d25b77e591ed42b08dbead4cce0921e7d4003e845638293c"}
Oct 30 02:53:58.951466 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:58.951438    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:59.096696 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:59.096650    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:59.243930 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:59.236926    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:59.413784 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:59.413749    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:53:59.415146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:59.415111    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 02:53:59.415281 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:59.415182    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 02:53:59.868246 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:53:59.868208    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:00.414484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:00.414447    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 02:54:00.414918 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:00.414512    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 02:54:01.080038 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:01.080002    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:54:01.414976 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:01.414883    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 02:54:01.414976 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:01.414956    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 02:54:01.489987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:01.489952    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:02.414473 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:02.414433    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 02:54:02.414657 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:02.414499    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 02:54:03.009922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:03.009889    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:03.414808 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:03.414773    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 30 02:54:03.414997 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:03.414826    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 30 02:54:04.418214 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:04.418185    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:54:04.419167 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:04.419146    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 02:54:04.651457 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:04.651422    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:05.308510 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:05.308464    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:06.361237 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:06.361203    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:07.191657 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.191615    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"]
Oct 30 02:54:07.194924 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.194899    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.218554 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.218512    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"openshift-service-ca.crt\""
Oct 30 02:54:07.218720 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.218588    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"default-dockercfg-28ksf\""
Oct 30 02:54:07.218794 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.218725    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"kube-root-ca.crt\""
Oct 30 02:54:07.220698 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.220678    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"]
Oct 30 02:54:07.221614 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.221599    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e11f03\"/\"simulator-w1-e11f03-kserve-self-signed-certs\""
Oct 30 02:54:07.236192 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.236166    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-home\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.236332 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.236312    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-dshm\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.236387 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.236368    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tls-certs\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.236428 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.236401    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tmp-dir\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.236428 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.236419    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ccq2\" (UniqueName: \"kubernetes.io/projected/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-kube-api-access-6ccq2\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.236548 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.236527    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-model-cache\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.269312 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.269288    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:07.337300 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.337262    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-dshm\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.337300 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.337299    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tls-certs\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.337532 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.337323    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tmp-dir\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.337532 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.337344    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ccq2\" (UniqueName: \"kubernetes.io/projected/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-kube-api-access-6ccq2\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.337532 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.337385    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-model-cache\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.337532 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.337442    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-home\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.337807 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.337777    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tmp-dir\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.337970 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.337816    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-home\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.337970 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.337861    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-model-cache\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.339636 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.339615    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-dshm\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.339928 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.339908    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tls-certs\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.354857 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.354821    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ccq2\" (UniqueName: \"kubernetes.io/projected/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-kube-api-access-6ccq2\") pod \"simulator-w1-e11f03-kserve-7794b65d8d-2kh6x\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") " pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.505982 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.505884    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:07.808141 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.808089    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"]
Oct 30 02:54:07.813257 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:54:07.813222    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a59ff6b_e6b4_459a_9b33_26088de3a5c9.slice/crio-a5190085f103bbab4c05eacca11f2330bca9016b701600e00efb242f05fe21ed WatchSource:0}: Error finding container a5190085f103bbab4c05eacca11f2330bca9016b701600e00efb242f05fe21ed: Status 404 returned error can't find the container with id a5190085f103bbab4c05eacca11f2330bca9016b701600e00efb242f05fe21ed
Oct 30 02:54:07.929114 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:07.929078    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:08.518239 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:08.518195    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" event={"ID":"6a59ff6b-e6b4-459a-9b33-26088de3a5c9","Type":"ContainerStarted","Data":"e0cb695ac54f015afed6b71160ad595937269bfa787166cd0f6679f91f4788d1"}
Oct 30 02:54:08.518665 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:08.518249    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" event={"ID":"6a59ff6b-e6b4-459a-9b33-26088de3a5c9","Type":"ContainerStarted","Data":"a5190085f103bbab4c05eacca11f2330bca9016b701600e00efb242f05fe21ed"}
Oct 30 02:54:08.554397 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:08.554351    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podStartSLOduration=1.55433864 podStartE2EDuration="1.55433864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:08.552631867 +0000 UTC m=+793.854810232" watchObservedRunningTime="2026-10-30 02:54:08.55433864 +0000 UTC m=+793.856517062"
Oct 30 02:54:09.506380 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:09.506344    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:09.526000 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:09.525973    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:09.903116 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:09.903086    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:11.307378 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:11.307332    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:12.903107 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:12.903069    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:13.771619 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.771586    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"]
Oct 30 02:54:13.774827 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.774795    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:13.798704 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.798682    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"openshift-service-ca.crt\""
Oct 30 02:54:13.798982 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.798965    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"kube-root-ca.crt\""
Oct 30 02:54:13.799083 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.799036    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"default-dockercfg-98cf4\""
Oct 30 02:54:13.799129 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.799111    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-810119\"/\"simulator-w2-810119-kserve-self-signed-certs\""
Oct 30 02:54:13.803458 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.803434    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"]
Oct 30 02:54:13.909091 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.909057    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tmp-dir\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:13.909091 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.909091    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tls-certs\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:13.909550 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.909215    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-home\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:13.909550 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.909266    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-model-cache\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:13.909550 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.909372    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88bml\" (UniqueName: \"kubernetes.io/projected/e076698c-e0eb-42d9-bdd9-2ecd3318da04-kube-api-access-88bml\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:13.909550 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.909408    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-dshm\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:13.982039 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:13.982003    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:14.010276 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.010251    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-model-cache\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.010407 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.010292    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bml\" (UniqueName: \"kubernetes.io/projected/e076698c-e0eb-42d9-bdd9-2ecd3318da04-kube-api-access-88bml\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.010484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.010419    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-dshm\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.010552 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.010523    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tmp-dir\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.010607 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.010553    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tls-certs\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.010665 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.010650    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-home\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.010718 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.010668    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-model-cache\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.010836 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.010816    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tmp-dir\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.010987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.010968    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-home\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.012670 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.012640    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-dshm\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.013139 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.013119    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tls-certs\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.027682 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.027632    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88bml\" (UniqueName: \"kubernetes.io/projected/e076698c-e0eb-42d9-bdd9-2ecd3318da04-kube-api-access-88bml\") pod \"simulator-w2-810119-kserve-6c8cdd7459-m22t4\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") " pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.087874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.087809    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:14.395655 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.395625    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"]
Oct 30 02:54:14.400025 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:54:14.399998    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode076698c_e0eb_42d9_bdd9_2ecd3318da04.slice/crio-effeade8bc2821d459ea5e08b79e7f49c3a38367bced3fd2fc167543b60fb3a3 WatchSource:0}: Error finding container effeade8bc2821d459ea5e08b79e7f49c3a38367bced3fd2fc167543b60fb3a3: Status 404 returned error can't find the container with id effeade8bc2821d459ea5e08b79e7f49c3a38367bced3fd2fc167543b60fb3a3
Oct 30 02:54:14.553400 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.553368    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" event={"ID":"e076698c-e0eb-42d9-bdd9-2ecd3318da04","Type":"ContainerStarted","Data":"e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d"}
Oct 30 02:54:14.553561 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.553414    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" event={"ID":"e076698c-e0eb-42d9-bdd9-2ecd3318da04","Type":"ContainerStarted","Data":"effeade8bc2821d459ea5e08b79e7f49c3a38367bced3fd2fc167543b60fb3a3"}
Oct 30 02:54:14.589674 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.589625    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podStartSLOduration=1.5896133730000002 podStartE2EDuration="1.589613373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:14.587502844 +0000 UTC m=+799.889681208" watchObservedRunningTime="2026-10-30 02:54:14.589613373 +0000 UTC m=+799.891791756"
Oct 30 02:54:14.636610 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:14.636583    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:15.089411 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:15.089384    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:16.201678 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:16.201648    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:17.506095 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:17.506062    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:17.518191 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:17.518166    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:17.580477 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:17.580452    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:54:17.685278 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:17.685240    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:19.376112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:19.376076    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:20.460467 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:20.460427    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:21.104510 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:21.104466    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:22.766677 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:22.766636    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:24.088931 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:24.088899    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:24.101398 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:24.101372    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:24.281178 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:24.281145    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:24.636812 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:24.636786    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:54:25.446379 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:54:25.446342    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:54:25.994037 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:25.994006    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:26.274545 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.274467    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"]
Oct 30 02:54:26.278540 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.278516    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.303621 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.303594    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-aeff1d\"/\"openshift-service-ca.crt\""
Oct 30 02:54:26.303806 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.303663    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-aeff1d\"/\"kube-root-ca.crt\""
Oct 30 02:54:26.303806 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.303716    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-aeff1d\"/\"premium-w0-aeff1d-kserve-self-signed-certs\""
Oct 30 02:54:26.303806 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.303716    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-aeff1d\"/\"default-dockercfg-gqwq9\""
Oct 30 02:54:26.305818 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.305794    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"]
Oct 30 02:54:26.355641 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.355617    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:26.446021 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.445998    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tmp-dir\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.446146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.446031    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-model-cache\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.446146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.446072    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvrnk\" (UniqueName: \"kubernetes.io/projected/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-kube-api-access-bvrnk\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.446212 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.446157    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-home\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.446246 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.446216    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tls-certs\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.446280 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.446246    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-dshm\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.547345 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.547312    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tmp-dir\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.547777 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.547358    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-model-cache\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.547777 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.547402    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvrnk\" (UniqueName: \"kubernetes.io/projected/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-kube-api-access-bvrnk\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.547777 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.547419    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-home\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.547777 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.547470    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tls-certs\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.547777 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.547511    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-dshm\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.548045 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.547824    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tmp-dir\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.548045 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.547901    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-model-cache\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.548045 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.547954    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-home\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.550612 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.550593    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-dshm\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.550734 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.550717    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tls-certs\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.565295 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.565271    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvrnk\" (UniqueName: \"kubernetes.io/projected/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-kube-api-access-bvrnk\") pod \"premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") " pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.593264 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.593240    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:26.887626 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:26.887552    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"]
Oct 30 02:54:26.891443 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:54:26.891417    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a7f6efd_1f7d_4d89_b020_0e6fb6d077bb.slice/crio-c9b857c779ae2b40331a325964a58541f51e59c4babc0fe6d091f415a68732ed WatchSource:0}: Error finding container c9b857c779ae2b40331a325964a58541f51e59c4babc0fe6d091f415a68732ed: Status 404 returned error can't find the container with id c9b857c779ae2b40331a325964a58541f51e59c4babc0fe6d091f415a68732ed
Oct 30 02:54:27.598793 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:27.598762    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:27.641563 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:27.641519    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" event={"ID":"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb","Type":"ContainerStarted","Data":"fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b"}
Oct 30 02:54:27.641563 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:27.641561    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" event={"ID":"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb","Type":"ContainerStarted","Data":"c9b857c779ae2b40331a325964a58541f51e59c4babc0fe6d091f415a68732ed"}
Oct 30 02:54:27.677624 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:27.677577    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podStartSLOduration=1.6775645080000001 podStartE2EDuration="1.677564508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:27.675977016 +0000 UTC m=+812.978155380" watchObservedRunningTime="2026-10-30 02:54:27.677564508 +0000 UTC m=+812.979742873"
Oct 30 02:54:28.593918 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:28.593883    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:29.059230 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:29.059194    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:30.661979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:30.661937    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:32.272795 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:32.272759    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:34.217053 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:34.217018    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:35.818861 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:35.818820    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:36.139145 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:36.139065    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:36.594383 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:36.594352    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:36.606625 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:36.606595    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:36.700181 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:36.700156    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:54:37.483389 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:37.483354    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:37.904944 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:37.904823    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:38.996156 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:38.996126    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:40.456673 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:40.456614    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:41.168414 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:41.168363    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:42.416768 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:42.416734    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:43.517610 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.517575    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"]
Oct 30 02:54:43.520738 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.520712    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.548868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.546969    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d9f3ac-openshift-default-dockercfg-97t95\""
Oct 30 02:54:43.548868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.548366    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"]
Oct 30 02:54:43.626774 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.626742    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.626951 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.626787    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.626951 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.626809    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.626951 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.626910    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.627071 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.627001    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bggf8\" (UniqueName: \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-kube-api-access-bggf8\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.627120 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.627074    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.627208 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.627189    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf52a76-996a-455b-8179-53780ef35d17-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.627277 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.627261    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.627326 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.627312    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf52a76-996a-455b-8179-53780ef35d17-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.728033 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.728000    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.728213 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.728055    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.728213 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.728087    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.728213 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.728116    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.728213 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.728152    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bggf8\" (UniqueName: \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-kube-api-access-bggf8\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.728445 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.728227    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.728445 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.728274    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf52a76-996a-455b-8179-53780ef35d17-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.728445 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.728335    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.728445 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.728362    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf52a76-996a-455b-8179-53780ef35d17-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.728640 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.728566    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.729321 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.728798    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.729455 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.729201    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf52a76-996a-455b-8179-53780ef35d17-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.729612 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.729588    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.729686 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.729601    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.732129 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.732098    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf52a76-996a-455b-8179-53780ef35d17-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.733095 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.733066    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.747193 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.747171    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.747345 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.747326    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bggf8\" (UniqueName: \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-kube-api-access-bggf8\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.834732 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.834701    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:43.864676 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:43.864636    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:44.151382 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:44.151301    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"]
Oct 30 02:54:44.155313 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:54:44.155284    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf52a76_996a_455b_8179_53780ef35d17.slice/crio-d6ca83edacb486f7aecd42b2ca06717006cb00e99e025fb7e2d32c97417e0761 WatchSource:0}: Error finding container d6ca83edacb486f7aecd42b2ca06717006cb00e99e025fb7e2d32c97417e0761: Status 404 returned error can't find the container with id d6ca83edacb486f7aecd42b2ca06717006cb00e99e025fb7e2d32c97417e0761
Oct 30 02:54:44.157577 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:44.157545    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:54:44.157647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:44.157611    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:54:44.157647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:44.157643    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:54:44.754737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:44.754694    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" event={"ID":"bbf52a76-996a-455b-8179-53780ef35d17","Type":"ContainerStarted","Data":"9649535fc8a99ea425e711695d514375a1d4c0acdb8f57569e5699c7b44aad71"}
Oct 30 02:54:44.755134 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:44.754752    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" event={"ID":"bbf52a76-996a-455b-8179-53780ef35d17","Type":"ContainerStarted","Data":"d6ca83edacb486f7aecd42b2ca06717006cb00e99e025fb7e2d32c97417e0761"}
Oct 30 02:54:44.813396 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:44.813351    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" podStartSLOduration=1.813338455 podStartE2EDuration="1.813338455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:44.802535747 +0000 UTC m=+830.104714115" watchObservedRunningTime="2026-10-30 02:54:44.813338455 +0000 UTC m=+830.115516819"
Oct 30 02:54:44.842004 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:44.841954    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:45.563995 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:45.563961    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:45.769666 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:45.769638    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:45.843925 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:45.842875    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 02:54:45.843925 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:45.842939    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 02:54:46.652964 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:46.652926    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:46.844871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:46.843950    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 02:54:46.844871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:46.844017    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 02:54:47.165377 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:47.165341    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:47.841871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:47.841210    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 02:54:47.841871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:47.841270    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 02:54:48.707234 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:48.707198    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:48.836871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:48.836189    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 02:54:48.836871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:48.836247    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 02:54:49.840039 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:49.840002    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 02:54:49.840641 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:49.840603    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 02:54:50.264918 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.264808    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:50.830301 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.830269    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"]
Oct 30 02:54:50.835225 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.835194    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:50.854744 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.854716    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"]
Oct 30 02:54:50.855190 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.854799    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f7d1a1\"/\"default-dockercfg-b9gn5\""
Oct 30 02:54:50.860903 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.860879    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f7d1a1\"/\"kube-root-ca.crt\""
Oct 30 02:54:50.861169 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.861150    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f7d1a1\"/\"openshift-service-ca.crt\""
Oct 30 02:54:50.915924 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.915883    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-model-cache\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:50.916098 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.916061    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-dshm\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:50.916167 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.916095    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czzkp\" (UniqueName: \"kubernetes.io/projected/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-kube-api-access-czzkp\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:50.916250 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.916232    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-home\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:50.916311 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.916281    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tmp-dir\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:50.916412 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.916396    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tls-certs\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:50.916957 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.916929    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:50.956489 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.956465    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f7d1a1\"/\"premium-w5-f7d1a1-kserve-self-signed-certs\""
Oct 30 02:54:50.969658 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.969624    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:50.970622 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.970598    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:50.971875 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:50.971838    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 02:54:51.019560 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.018153    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tmp-dir\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.019560 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.018261    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tls-certs\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.019560 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.018342    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-model-cache\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.019560 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.018507    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-dshm\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.019560 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.018538    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czzkp\" (UniqueName: \"kubernetes.io/projected/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-kube-api-access-czzkp\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.019560 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.018595    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-home\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.019560 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.019076    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-home\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.019560 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.019340    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tmp-dir\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.025908 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.025290    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-model-cache\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.042263 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.042233    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tls-certs\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.043770 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.043712    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-dshm\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.061163 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.061129    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czzkp\" (UniqueName: \"kubernetes.io/projected/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-kube-api-access-czzkp\") pod \"premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") " pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.149128 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.149051    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:51.447003 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.446924    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"]
Oct 30 02:54:51.450709 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:54:51.450679    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podede5a7fc_13b5_40ea_a716_7b0acdaacc93.slice/crio-3fd2df603924cac52bf5a3587184f88078163d1c19cb2bd1a610796fdf7457d1 WatchSource:0}: Error finding container 3fd2df603924cac52bf5a3587184f88078163d1c19cb2bd1a610796fdf7457d1: Status 404 returned error can't find the container with id 3fd2df603924cac52bf5a3587184f88078163d1c19cb2bd1a610796fdf7457d1
Oct 30 02:54:51.811208 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.811169    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" event={"ID":"ede5a7fc-13b5-40ea-a716-7b0acdaacc93","Type":"ContainerStarted","Data":"ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924"}
Oct 30 02:54:51.811208 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.811214    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" event={"ID":"ede5a7fc-13b5-40ea-a716-7b0acdaacc93","Type":"ContainerStarted","Data":"3fd2df603924cac52bf5a3587184f88078163d1c19cb2bd1a610796fdf7457d1"}
Oct 30 02:54:51.845267 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.845198    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:51.848782 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:51.848728    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podStartSLOduration=1.848715602 podStartE2EDuration="1.848715602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:51.846418095 +0000 UTC m=+837.148596459" watchObservedRunningTime="2026-10-30 02:54:51.848715602 +0000 UTC m=+837.150893966"
Oct 30 02:54:52.149962 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:52.149839    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:54:53.679388 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:53.679358    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:55.180385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:55.180345    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:55.445274 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:54:55.445191    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:54:56.795577 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:56.795545    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:57.359490 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:57.359455    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:58.547038 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:54:58.547001    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:00.191961 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:00.191928    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:01.149573 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:01.149536    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:55:01.162608 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:01.162585    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:55:01.875561 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:01.875528    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:55:02.129270 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:02.129195    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:03.981675 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:03.981629    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:05.995369 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:05.995266    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:07.657141 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:07.657109    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:09.248255 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:09.248225    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:11.007767 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:11.007725    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:12.582223 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:12.582183    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:14.315969 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.315839    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"]
Oct 30 02:55:14.321676 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.321609    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.338301 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.338232    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7d1788-openshift-default-dockercfg-29c4p\""
Oct 30 02:55:14.352943 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.352872    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"]
Oct 30 02:55:14.395625 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.395579    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:14.482701 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.482667    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.482701 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.482707    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.482974 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.482736    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.482974 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.482926    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrbs\" (UniqueName: \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-kube-api-access-jgrbs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.483087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.483063    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.483146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.483104    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.483146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.483136    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.483257 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.483242    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.483302 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.483266    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.584857 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.584765    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrbs\" (UniqueName: \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-kube-api-access-jgrbs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585007 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.584893    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585007 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.584915    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585007 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.584936    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585007 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.584989    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585185 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.585007    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585185 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.585033    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585185 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.585056    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585185 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.585087    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.585361    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.585377    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585506 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.585485    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585720 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.585699    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.585856 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.585705    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.587469 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.587440    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.587768 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.587749    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.602566 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.602542    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.602732 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.602714    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrbs\" (UniqueName: \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-kube-api-access-jgrbs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.646898 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.646873    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:14.959308 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.959226    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"]
Oct 30 02:55:14.963326 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:55:14.963303    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d65367_9c08_4b32_b9d4_002d9bc4c55c.slice/crio-b34acf3757968948896430814444c10c6c1cc8462919f6997dce3e3ebc9ec7ea WatchSource:0}: Error finding container b34acf3757968948896430814444c10c6c1cc8462919f6997dce3e3ebc9ec7ea: Status 404 returned error can't find the container with id b34acf3757968948896430814444c10c6c1cc8462919f6997dce3e3ebc9ec7ea
Oct 30 02:55:14.965424 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.965394    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:55:14.965504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.965455    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:55:14.965504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:14.965482    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:55:15.949755 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:15.949719    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" event={"ID":"f8d65367-9c08-4b32-b9d4-002d9bc4c55c","Type":"ContainerStarted","Data":"ea737c0c4e50cc84f31640379f6929ce6464953dd1240fdcad91fac0417caf74"}
Oct 30 02:55:15.950166 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:15.949775    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" event={"ID":"f8d65367-9c08-4b32-b9d4-002d9bc4c55c","Type":"ContainerStarted","Data":"b34acf3757968948896430814444c10c6c1cc8462919f6997dce3e3ebc9ec7ea"}
Oct 30 02:55:15.982161 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:15.982129    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:16.000245 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:16.000195    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" podStartSLOduration=2.000179174 podStartE2EDuration="2.000179174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:15.998183038 +0000 UTC m=+861.300361402" watchObservedRunningTime="2026-10-30 02:55:16.000179174 +0000 UTC m=+861.302357568"
Oct 30 02:55:16.647238 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:16.647182    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:16.649411 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:16.649359    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:55:16.649598 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:16.649475    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 02:55:17.652873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:17.652273    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:55:17.652873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:17.652336    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 02:55:17.762872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:17.758838    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:18.647737 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:18.647694    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:55:18.647974 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:18.647747    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 02:55:19.617709 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:19.617673    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:19.650389 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:19.650344    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:55:19.650652 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:19.650408    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 02:55:20.651244 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:20.651209    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:55:20.651819 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:20.651785    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 02:55:21.093411 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:21.093337    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:21.649333 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:21.649298    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 02:55:21.649643 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:21.649603    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 02:55:22.813738 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:22.813681    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:23.210255 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:23.210161    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:23.211125 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:23.211029    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:23.212251 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:23.212231    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 02:55:24.455175 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:24.455147    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:25.445908 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:55:25.445868    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:55:26.009020 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:26.008989    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:27.464412 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:27.464374    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:29.487606 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:29.487576    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:31.330192 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:31.330153    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:33.673387 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:33.673352    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:35.957268 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:35.957238    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:37.779923 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:37.779887    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:39.407981 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:39.407950    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:41.079299 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:41.079264    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:42.638072 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:42.638034    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:44.069587 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.069551    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"]
Oct 30 02:55:44.074122 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.074100    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.087951 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.087928    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-aeff1d-openshift-default-dockercfg-8cpz8\""
Oct 30 02:55:44.107632 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.107589    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"]
Oct 30 02:55:44.117566 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.117527    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:44.186633 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.186591    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.186828 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.186646    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.186828 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.186679    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpcdw\" (UniqueName: \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-kube-api-access-gpcdw\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.186828 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.186770    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2c026e-88e1-4e26-abe7-889ded07492c-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.187109 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.186945    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.187109 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.186987    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.187109 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.187017    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.187109 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.187048    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.187109 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.187090    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.288602 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.288560    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.288602 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.288616    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.288916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.288646    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpcdw\" (UniqueName: \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-kube-api-access-gpcdw\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.288916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.288719    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2c026e-88e1-4e26-abe7-889ded07492c-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.288916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.288833    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.288916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.288886    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.288916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.288916    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.289211 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.288948    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.289211 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.288992    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.290444 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.290408    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2c026e-88e1-4e26-abe7-889ded07492c-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.291191 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.291089    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.291398 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.291357    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.291506 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.291470    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.291805 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.291778    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.294978 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.294951    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.315837 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.313489    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.315837 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.314224    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpcdw\" (UniqueName: \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-kube-api-access-gpcdw\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.315837 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.315079    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.387679 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.387577    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:44.825995 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.818587    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"]
Oct 30 02:55:44.987227 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.987191    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:55:44.987494 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.987469    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:55:44.987682 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:44.987661    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:55:45.212540 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:45.212501    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" event={"ID":"4a2c026e-88e1-4e26-abe7-889ded07492c","Type":"ContainerStarted","Data":"7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9"}
Oct 30 02:55:45.213017 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:45.212559    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" event={"ID":"4a2c026e-88e1-4e26-abe7-889ded07492c","Type":"ContainerStarted","Data":"dd6dced5eef01f6751fff5f628cddcb5f8f2ed9ad106fde93ebd4c0954e2372f"}
Oct 30 02:55:45.250444 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:45.250392    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" podStartSLOduration=1.250378547 podStartE2EDuration="1.250378547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:45.249266114 +0000 UTC m=+890.551444480" watchObservedRunningTime="2026-10-30 02:55:45.250378547 +0000 UTC m=+890.552556906"
Oct 30 02:55:45.388301 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:45.388219    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:45.892735 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:45.892701    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:46.390154 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.390118    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:55:46.390639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.390208    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 02:55:46.414476 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.414442    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:46.470465 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.470429    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"]
Oct 30 02:55:46.474502 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.474483    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.500514 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.500482    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f7d1a1\"/\"distinct-w5-f7d1a1-kserve-self-signed-certs\""
Oct 30 02:55:46.503976 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.503955    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"]
Oct 30 02:55:46.547365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.547328    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:46.549650 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.549625    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-tmp-dir\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.549794 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.549661    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcm6l\" (UniqueName: \"kubernetes.io/projected/ec889f00-32b6-4c2c-931e-f42c26a0026c-kube-api-access-pcm6l\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.549895 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.549819    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-model-cache\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.549966 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.549941    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-home\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.550021 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.550003    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-dshm\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.550102 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.550086    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec889f00-32b6-4c2c-931e-f42c26a0026c-tls-certs\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.651090 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.651008    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-tmp-dir\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.651090 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.651050    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcm6l\" (UniqueName: \"kubernetes.io/projected/ec889f00-32b6-4c2c-931e-f42c26a0026c-kube-api-access-pcm6l\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.651292 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.651099    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-model-cache\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.651292 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.651152    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-home\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.651292 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.651191    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-dshm\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.651292 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.651259    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec889f00-32b6-4c2c-931e-f42c26a0026c-tls-certs\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.651555 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.651466    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-tmp-dir\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.651611 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.651557    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-model-cache\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.651611 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.651571    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-home\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.653633 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.653605    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-dshm\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.653749 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.653673    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec889f00-32b6-4c2c-931e-f42c26a0026c-tls-certs\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.668220 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.668193    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcm6l\" (UniqueName: \"kubernetes.io/projected/ec889f00-32b6-4c2c-931e-f42c26a0026c-kube-api-access-pcm6l\") pod \"distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:46.787600 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:46.787566    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:47.111720 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:47.111682    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"]
Oct 30 02:55:47.116527 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:55:47.116486    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec889f00_32b6_4c2c_931e_f42c26a0026c.slice/crio-ad5560e2eeb7bd8574b71ee5f2ad2ae13ec38545fdb5bf22fdad8a9de788cccd WatchSource:0}: Error finding container ad5560e2eeb7bd8574b71ee5f2ad2ae13ec38545fdb5bf22fdad8a9de788cccd: Status 404 returned error can't find the container with id ad5560e2eeb7bd8574b71ee5f2ad2ae13ec38545fdb5bf22fdad8a9de788cccd
Oct 30 02:55:47.250930 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:47.250891    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" event={"ID":"ec889f00-32b6-4c2c-931e-f42c26a0026c","Type":"ContainerStarted","Data":"65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187"}
Oct 30 02:55:47.250930 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:47.250937    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" event={"ID":"ec889f00-32b6-4c2c-931e-f42c26a0026c","Type":"ContainerStarted","Data":"ad5560e2eeb7bd8574b71ee5f2ad2ae13ec38545fdb5bf22fdad8a9de788cccd"}
Oct 30 02:55:47.288367 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:47.288312    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podStartSLOduration=1.288298275 podStartE2EDuration="1.288298275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:47.286388629 +0000 UTC m=+892.588566997" watchObservedRunningTime="2026-10-30 02:55:47.288298275 +0000 UTC m=+892.590476679"
Oct 30 02:55:47.388715 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:47.388628    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:55:47.388715 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:47.388679    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 02:55:47.788658 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:47.788576    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:48.180826 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:48.180793    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:48.387901 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:48.387869    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:55:48.388087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:48.387923    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 02:55:49.388175 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:49.388141    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:55:49.388582 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:49.388194    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 02:55:49.907888 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:49.907844    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:50.388031 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:50.387998    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:55:50.388202 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:50.388053    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 02:55:51.388831 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:51.388795    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 30 02:55:51.389273 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:51.388891    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 30 02:55:51.617488 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:51.617450    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:52.393209 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:52.393181    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:52.393708 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:52.393558    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:52.394474 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:52.394452    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 02:55:53.139992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:53.139958    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:54.782104 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:54.782021    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:54.928480 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:54.928440    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:55:54.931832 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:54.931802    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:55:54.932013 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:54.931969    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 02:55:54.935248 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:54.935226    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 02:55:55.445066 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:55:55.445031    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:55:56.017920 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:56.017885    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:56.324738 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:56.324710    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:56.788199 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:56.788121    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:56.800679 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:56.800653    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:57.319246 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:57.319213    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:55:57.801268 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:55:57.801232    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:00.464174 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:00.464137    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:02.149192 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:02.149151    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:04.012126 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.012088    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:04.270381 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.267824    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"]
Oct 30 02:56:04.273799 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.273771    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.287141 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.287112    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"kube-root-ca.crt\""
Oct 30 02:56:04.287281 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.287253    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"openshift-service-ca.crt\""
Oct 30 02:56:04.287398 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.287380    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"default-dockercfg-4fftw\""
Oct 30 02:56:04.287514 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.287445    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d9f3ac\"/\"distinct-w3-d9f3ac-kserve-self-signed-certs\""
Oct 30 02:56:04.288691 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.288658    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"]
Oct 30 02:56:04.343905 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.343869    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:04.367277 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.367252    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-model-cache\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.367399 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.367294    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-dshm\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.367399 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.367314    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-home\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.367480 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.367436    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tls-certs\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.367535 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.367521    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tmp-dir\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.367572 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.367548    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6gl\" (UniqueName: \"kubernetes.io/projected/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-kube-api-access-nk6gl\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.468725 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.468694    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-dshm\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.468934 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.468735    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-home\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.468934 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.468780    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tls-certs\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.468934 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.468876    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tmp-dir\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.468934 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.468902    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6gl\" (UniqueName: \"kubernetes.io/projected/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-kube-api-access-nk6gl\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.469171 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.469029    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-model-cache\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.469297 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.469270    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-home\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.469402 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.469371    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-model-cache\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.469499 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.469485    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tmp-dir\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.471537 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.471505    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-dshm\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.472275 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.472253    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tls-certs\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.490108 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.490076    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6gl\" (UniqueName: \"kubernetes.io/projected/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-kube-api-access-nk6gl\") pod \"distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") " pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.588113 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.588080    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:04.885779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:04.885702    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"]
Oct 30 02:56:04.889879 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:56:04.889836    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec6871e6_84d2_42fb_b7b2_060e5f3fcc8b.slice/crio-1e2990bd37be7a0f0d7ed4c02044e7f3bf313d4a6978fee29d251ad4e6cd2e29 WatchSource:0}: Error finding container 1e2990bd37be7a0f0d7ed4c02044e7f3bf313d4a6978fee29d251ad4e6cd2e29: Status 404 returned error can't find the container with id 1e2990bd37be7a0f0d7ed4c02044e7f3bf313d4a6978fee29d251ad4e6cd2e29
Oct 30 02:56:05.381465 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.381429    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" event={"ID":"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b","Type":"ContainerStarted","Data":"188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d"}
Oct 30 02:56:05.381906 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.381475    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" event={"ID":"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b","Type":"ContainerStarted","Data":"1e2990bd37be7a0f0d7ed4c02044e7f3bf313d4a6978fee29d251ad4e6cd2e29"}
Oct 30 02:56:05.417277 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.417223    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podStartSLOduration=1.417208638 podStartE2EDuration="1.417208638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:05.416747443 +0000 UTC m=+910.718925807" watchObservedRunningTime="2026-10-30 02:56:05.417208638 +0000 UTC m=+910.719387002"
Oct 30 02:56:05.588427 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.588378    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:05.693102 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.693021    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:05.955233 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.955156    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"]
Oct 30 02:56:05.959027 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.959010    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:05.983063 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.983034    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-aeff1d\"/\"distinct-w0-aeff1d-kserve-self-signed-certs\""
Oct 30 02:56:05.986596 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.986572    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-model-cache\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:05.986763 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.986603    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-dshm\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:05.986763 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.986623    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6e7199-66a6-4043-9bdd-ca245968dfef-tls-certs\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:05.986913 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.986889    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gksxr\" (UniqueName: \"kubernetes.io/projected/4f6e7199-66a6-4043-9bdd-ca245968dfef-kube-api-access-gksxr\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:05.986970 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.986956    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-tmp-dir\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:05.987076 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.987058    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-home\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:05.987398 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:05.987383    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"]
Oct 30 02:56:06.055706 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.055677    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:06.088473 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.088445    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-model-cache\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.088607 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.088476    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-dshm\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.088607 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.088504    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6e7199-66a6-4043-9bdd-ca245968dfef-tls-certs\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.088674 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.088634    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gksxr\" (UniqueName: \"kubernetes.io/projected/4f6e7199-66a6-4043-9bdd-ca245968dfef-kube-api-access-gksxr\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.088715 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.088684    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-tmp-dir\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.088755 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.088744    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-home\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.088971 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.088948    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-model-cache\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.089142 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.089121    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-tmp-dir\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.089183 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.089145    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-home\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.090784 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.090765    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-dshm\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.091183 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.091164    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6e7199-66a6-4043-9bdd-ca245968dfef-tls-certs\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.105991 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.105964    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gksxr\" (UniqueName: \"kubernetes.io/projected/4f6e7199-66a6-4043-9bdd-ca245968dfef-kube-api-access-gksxr\") pod \"distinct-w0-aeff1d-kserve-6655df77c4-kctg9\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") " pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.272710 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.272626    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:06.572313 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:06.572280    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"]
Oct 30 02:56:06.576527 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:56:06.576493    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f6e7199_66a6_4043_9bdd_ca245968dfef.slice/crio-b61cf741dbaeb4667d730de0a130af07b9eeed9a9f53ce872188970a27d16b75 WatchSource:0}: Error finding container b61cf741dbaeb4667d730de0a130af07b9eeed9a9f53ce872188970a27d16b75: Status 404 returned error can't find the container with id b61cf741dbaeb4667d730de0a130af07b9eeed9a9f53ce872188970a27d16b75
Oct 30 02:56:07.394767 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:07.394720    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" event={"ID":"4f6e7199-66a6-4043-9bdd-ca245968dfef","Type":"ContainerStarted","Data":"67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa"}
Oct 30 02:56:07.394992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:07.394781    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" event={"ID":"4f6e7199-66a6-4043-9bdd-ca245968dfef","Type":"ContainerStarted","Data":"b61cf741dbaeb4667d730de0a130af07b9eeed9a9f53ce872188970a27d16b75"}
Oct 30 02:56:07.430416 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:07.430353    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podStartSLOduration=2.430338164 podStartE2EDuration="2.430338164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:07.429483391 +0000 UTC m=+912.731661755" watchObservedRunningTime="2026-10-30 02:56:07.430338164 +0000 UTC m=+912.732516530"
Oct 30 02:56:07.575283 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:07.575249    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:08.272710 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:08.272677    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:09.016317 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:09.016269    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:10.732262 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:10.732222    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:12.589316 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:12.589239    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:14.010431 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.010394    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:14.174949 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.174916    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:14.589163 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.589129    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:14.601546 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.601524    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:14.702112 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.702084    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"]
Oct 30 02:56:14.706074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.706052    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.728436 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.728414    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"]
Oct 30 02:56:14.786121 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.786093    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.786248 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.786154    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.786332 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.786310    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.786369 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.786341    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.786369 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.786358    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.786436 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.786407    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.786436 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.786431    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.786498 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.786461    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69pvp\" (UniqueName: \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-kube-api-access-69pvp\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.786553 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.786534    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888066 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.887978    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888066 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888016    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888066 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888039    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888332 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888074    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888332 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888094    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888332 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888122    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69pvp\" (UniqueName: \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-kube-api-access-69pvp\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888332 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888310    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888542 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888371    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888542 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888438    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888542 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888451    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888542 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888470    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888742 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888578    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888742 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888701    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.888947 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.888922    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.890714 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.890691    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.890873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.890840    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.905425 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.905387    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:14.905659 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:14.905641    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69pvp\" (UniqueName: \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-kube-api-access-69pvp\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:15.019298 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:15.019264    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:15.332169 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:15.332134    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"]
Oct 30 02:56:15.335752 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:56:15.335725    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7983d72_b33b_4849_94ac_58c2a0e8fbc7.slice/crio-3208a8f1cc31f1353de3c84876aafbff6eda0a7cce36eee5806baaacd49220a6 WatchSource:0}: Error finding container 3208a8f1cc31f1353de3c84876aafbff6eda0a7cce36eee5806baaacd49220a6: Status 404 returned error can't find the container with id 3208a8f1cc31f1353de3c84876aafbff6eda0a7cce36eee5806baaacd49220a6
Oct 30 02:56:15.338099 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:15.338069    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:15.338187 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:15.338130    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:15.338187 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:15.338178    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:15.438542 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:15.438511    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" event={"ID":"f7983d72-b33b-4849-94ac-58c2a0e8fbc7","Type":"ContainerStarted","Data":"adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2"}
Oct 30 02:56:15.438685 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:15.438568    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" event={"ID":"f7983d72-b33b-4849-94ac-58c2a0e8fbc7","Type":"ContainerStarted","Data":"3208a8f1cc31f1353de3c84876aafbff6eda0a7cce36eee5806baaacd49220a6"}
Oct 30 02:56:15.450232 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:15.450206    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:15.475696 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:15.475641    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podStartSLOduration=1.475623538 podStartE2EDuration="1.475623538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:15.475039522 +0000 UTC m=+920.777217901" watchObservedRunningTime="2026-10-30 02:56:15.475623538 +0000 UTC m=+920.777801904"
Oct 30 02:56:16.013327 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:16.013290    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:16.021157 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:16.021132    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:16.040064 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:16.040009    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:56:16.040341 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:16.040306    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 02:56:16.273830 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:16.273728    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:16.286105 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:16.286074    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:16.459260 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:16.459226    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:17.031870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:17.031213    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:56:17.031870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:17.031275    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 02:56:17.864088 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:17.864058    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:18.037872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:18.037241    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:56:18.037872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:18.037305    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 02:56:19.034092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:19.033617    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:56:19.034092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:19.033682    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 02:56:19.808635 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:19.808603    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:20.026872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:20.024224    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:56:20.026872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:20.024290    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 02:56:21.032886 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:21.030258    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 02:56:21.032886 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:21.030321    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 02:56:21.722365 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:21.722326    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:22.024519 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:22.024430    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:22.024754 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:22.024729    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:22.025640 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:22.025616    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 02:56:23.292705 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:23.292670    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:24.822894 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:24.822832    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:25.509282 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:56:25.509243    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:56:27.912186 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:27.912154    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:28.951604 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:28.951569    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"]
Oct 30 02:56:28.956339 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:28.956320    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:28.981777 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:28.981750    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f7d1a1\"/\"distinct-2-w5-f7d1a1-kserve-self-signed-certs\""
Oct 30 02:56:28.984319 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:28.984296    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"]
Oct 30 02:56:29.035498 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.035468    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:29.133760 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.133727    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c837855-e677-46af-91b9-2f2a06306d5b-tls-certs\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.133967 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.133880    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2thm\" (UniqueName: \"kubernetes.io/projected/2c837855-e677-46af-91b9-2f2a06306d5b-kube-api-access-z2thm\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.133967 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.133905    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-dshm\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.133967 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.133925    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-model-cache\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.134076 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.134059    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-tmp-dir\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.134122 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.134094    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-home\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.234900 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.234796    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-tmp-dir\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.234900 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.234833    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-home\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.235137 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.234943    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c837855-e677-46af-91b9-2f2a06306d5b-tls-certs\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.235137 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.235030    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2thm\" (UniqueName: \"kubernetes.io/projected/2c837855-e677-46af-91b9-2f2a06306d5b-kube-api-access-z2thm\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.235137 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.235064    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-dshm\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.235137 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.235089    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-model-cache\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.235351 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.235287    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-tmp-dir\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.235351 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.235337    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-home\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.235525 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.235497    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-model-cache\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.237642 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.237615    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c837855-e677-46af-91b9-2f2a06306d5b-tls-certs\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.238084 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.238064    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-dshm\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.252285 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.252264    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2thm\" (UniqueName: \"kubernetes.io/projected/2c837855-e677-46af-91b9-2f2a06306d5b-kube-api-access-z2thm\") pod \"distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") " pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.271065 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.271030    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:29.596139 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.596102    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"]
Oct 30 02:56:29.599317 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:56:29.599286    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c837855_e677_46af_91b9_2f2a06306d5b.slice/crio-5010bdf4c6f05458c898df586e39cb6218414f4f85ede8be4c3fdc1398701bd0 WatchSource:0}: Error finding container 5010bdf4c6f05458c898df586e39cb6218414f4f85ede8be4c3fdc1398701bd0: Status 404 returned error can't find the container with id 5010bdf4c6f05458c898df586e39cb6218414f4f85ede8be4c3fdc1398701bd0
Oct 30 02:56:29.730196 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:29.729577    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:30.569156 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:30.569105    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" event={"ID":"2c837855-e677-46af-91b9-2f2a06306d5b","Type":"ContainerStarted","Data":"523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93"}
Oct 30 02:56:30.569634 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:30.569164    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" event={"ID":"2c837855-e677-46af-91b9-2f2a06306d5b","Type":"ContainerStarted","Data":"5010bdf4c6f05458c898df586e39cb6218414f4f85ede8be4c3fdc1398701bd0"}
Oct 30 02:56:30.604151 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:30.604095    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podStartSLOduration=2.604080966 podStartE2EDuration="2.604080966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:30.603480366 +0000 UTC m=+935.905658730" watchObservedRunningTime="2026-10-30 02:56:30.604080966 +0000 UTC m=+935.906259330"
Oct 30 02:56:31.271511 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:31.271484    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:31.278521 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:31.278493    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:32.018644 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:32.018612    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:32.134383 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:32.134347    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:32.920063 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:32.920035    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:34.574508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:34.574472    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:36.395745 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:36.395655    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:38.085905 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:38.085863    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:39.271284 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:39.271248    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:39.283434 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:39.283408    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:39.633967 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:39.633936    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:39.911747 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:39.911637    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:40.032054 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:40.032023    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:40.201504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:40.201419    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:41.407280 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:41.407225    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:42.990069 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:42.990031    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:43.900872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.900806    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"]
Oct 30 02:56:43.905173 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.905155    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:43.927227 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.927204    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"]
Oct 30 02:56:43.936198 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.936175    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"]
Oct 30 02:56:43.939916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.939896    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:43.961184 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.961160    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"]
Oct 30 02:56:43.990998 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.990954    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:43.991325 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991009    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:43.991325 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991134    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:43.991325 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991171    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3aae8e9-b846-4409-877b-618c0b69302b-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:43.991325 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991204    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:43.991325 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991248    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:43.991488 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991358    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:43.991488 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991381    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:43.991488 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991407    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:43.991488 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991466    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:43.991626 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991491    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zccgz\" (UniqueName: \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-kube-api-access-zccgz\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:43.991626 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991529    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:43.991626 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991548    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ctcf\" (UniqueName: \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-kube-api-access-8ctcf\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:43.991626 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991569    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:43.991820 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991753    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3aae8e9-b846-4409-877b-618c0b69302b-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:43.991898 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991814    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:43.991898 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991861    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:43.992011 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:43.991907    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c5f62d-44a9-4d43-aff1-6f0191511107-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.092929 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.092900    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.093125 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.092938    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093125 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.092972    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c5f62d-44a9-4d43-aff1-6f0191511107-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093125 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093003    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093125 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093031    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.093125 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093064    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093125 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093086    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3aae8e9-b846-4409-877b-618c0b69302b-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.093438 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093207    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.093438 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093251    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093438 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093298    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.093438 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093332    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093438 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093365    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.093438 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093381    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093438 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093434    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093805 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093461    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zccgz\" (UniqueName: \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-kube-api-access-zccgz\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.093805 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093503    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093805 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093528    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ctcf\" (UniqueName: \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-kube-api-access-8ctcf\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093805 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093559    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.093805 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093659    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093805 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093676    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.093805 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093726    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3aae8e9-b846-4409-877b-618c0b69302b-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.094182 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093807    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3aae8e9-b846-4409-877b-618c0b69302b-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.094182 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093968    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.094182 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.093524    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.094182 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.094099    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-data\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.094435 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.094293    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-credential-socket\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.094435 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.094398    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-certs\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.094909 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.094883    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c5f62d-44a9-4d43-aff1-6f0191511107-istiod-ca-cert\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.095813 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.095783    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.096350 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.096332    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3aae8e9-b846-4409-877b-618c0b69302b-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.096647 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.096626    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-envoy\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.096895 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.096877    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-podinfo\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.110699 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.110669    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zccgz\" (UniqueName: \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-kube-api-access-zccgz\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.110816 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.110798    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ctcf\" (UniqueName: \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-kube-api-access-8ctcf\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.110917 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.110887    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.110979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.110964    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-token\") pod \"e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") " pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.219193 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.219105    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:44.222642 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.222616    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"]
Oct 30 02:56:44.226346 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.226328    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.249991 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.249962    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"]
Oct 30 02:56:44.255439 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.255414    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"]
Oct 30 02:56:44.257126 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.257100    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:44.260922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.260901    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.280053 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.280027    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"]
Oct 30 02:56:44.296272 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.296247    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrbsk\" (UniqueName: \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-kube-api-access-mrbsk\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.296375 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.296321    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.296375 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.296358    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.296487 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.296426    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.296487 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.296472    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.296633 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.296526    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.296633 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.296580    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f8a79d-092c-443e-904a-7de32cd1af90-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.296633 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.296610    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.296781 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.296688    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f8a79d-092c-443e-904a-7de32cd1af90-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.398260 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398230    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.398408 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398276    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.398408 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398306    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.398408 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398372    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.398408 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398392    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.398639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398416    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.398639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398435    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.398639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398476    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f8a79d-092c-443e-904a-7de32cd1af90-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.398639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398494    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.398639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398532    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f8a79d-092c-443e-904a-7de32cd1af90-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.398639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398591    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.398639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398614    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.398639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398615    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.398639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.398632    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlsnr\" (UniqueName: \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-kube-api-access-hlsnr\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.399116 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.399006    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrbsk\" (UniqueName: \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-kube-api-access-mrbsk\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.399116 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.399035    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.399116 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.399078    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.399116 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.399101    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.399331 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.399119    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.399331 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.399121    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.400316 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.399625    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.400316 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.399797    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.400316 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.400273    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f8a79d-092c-443e-904a-7de32cd1af90-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.401204 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.401185    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.401485 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.401467    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f8a79d-092c-443e-904a-7de32cd1af90-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.416050 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.416031    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.417159 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.417139    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrbsk\" (UniqueName: \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-kube-api-access-mrbsk\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.500058 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500006    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500058 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500036    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500186 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500066    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500276 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500189    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500276 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500226    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500276 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500253    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlsnr\" (UniqueName: \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-kube-api-access-hlsnr\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500433 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500356    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500433 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500415    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500567 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500488    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500567 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500540    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500567 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500560    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-credential-socket\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500798 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500778    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-certs\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.500798 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.500792    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-data\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.501034 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.501013    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istiod-ca-cert\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.502415 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.502398    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-podinfo\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.502459 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.502420    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-envoy\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.517894 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.517874    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-token\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.518028 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.518012    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlsnr\" (UniqueName: \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-kube-api-access-hlsnr\") pod \"e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") " pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.551196 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.551172    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"]
Oct 30 02:56:44.555594 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:56:44.555571    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3aae8e9_b846_4409_877b_618c0b69302b.slice/crio-854492638e4ede0ad357773fd78b742947eddb62ed64b60f615e014c7fe3dd92 WatchSource:0}: Error finding container 854492638e4ede0ad357773fd78b742947eddb62ed64b60f615e014c7fe3dd92: Status 404 returned error can't find the container with id 854492638e4ede0ad357773fd78b742947eddb62ed64b60f615e014c7fe3dd92
Oct 30 02:56:44.557992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.557957    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:44.558088 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.558023    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:44.558088 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.558050    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:44.587635 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.587616    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:44.597731 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.597708    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:44.597841 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.597773    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"]
Oct 30 02:56:44.603171 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:56:44.603147    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c5f62d_44a9_4d43_aff1_6f0191511107.slice/crio-1606eadb52ba0f78b975115af3e81a44ff7e986dec857e383d0aff95a801b08f WatchSource:0}: Error finding container 1606eadb52ba0f78b975115af3e81a44ff7e986dec857e383d0aff95a801b08f: Status 404 returned error can't find the container with id 1606eadb52ba0f78b975115af3e81a44ff7e986dec857e383d0aff95a801b08f
Oct 30 02:56:44.611962 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.611933    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:44.612048 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.611993    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:44.612048 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.612023    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:44.655128 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.650729    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:44.658977 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.657363    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" event={"ID":"f3aae8e9-b846-4409-877b-618c0b69302b","Type":"ContainerStarted","Data":"017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c"}
Oct 30 02:56:44.658977 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.657413    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" event={"ID":"f3aae8e9-b846-4409-877b-618c0b69302b","Type":"ContainerStarted","Data":"854492638e4ede0ad357773fd78b742947eddb62ed64b60f615e014c7fe3dd92"}
Oct 30 02:56:44.659992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.659895    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" event={"ID":"17c5f62d-44a9-4d43-aff1-6f0191511107","Type":"ContainerStarted","Data":"1606eadb52ba0f78b975115af3e81a44ff7e986dec857e383d0aff95a801b08f"}
Oct 30 02:56:44.702877 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.700317    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podStartSLOduration=1.700298202 podStartE2EDuration="1.700298202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:44.697169581 +0000 UTC m=+949.999347949" watchObservedRunningTime="2026-10-30 02:56:44.700298202 +0000 UTC m=+950.002476570"
Oct 30 02:56:44.813924 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.813876    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"]
Oct 30 02:56:44.819206 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.819182    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:44.842513 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.842348    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"]
Oct 30 02:56:44.909703 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.909663    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:44.909913 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.909718    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:44.909913 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.909830    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:44.909913 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.909866    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfa511b-667c-4f96-8509-785691b2cb17-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:44.909913 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.909889    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqh8s\" (UniqueName: \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-kube-api-access-fqh8s\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:44.910157 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.909985    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:44.910157 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.910050    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfa511b-667c-4f96-8509-785691b2cb17-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:44.910157 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.910103    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:44.910317 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.910164    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:44.925527 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.925492    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"]
Oct 30 02:56:44.937517 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.936255    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:44.937517 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.936340    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:44.937517 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.936411    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:44.937627 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.937510    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"]
Oct 30 02:56:44.940037 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:56:44.940014    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f8a79d_092c_443e_904a_7de32cd1af90.slice/crio-517447bf2c88c68b573b4900652ef6f0d407d7639f5e165790a7805b115f4bfd WatchSource:0}: Error finding container 517447bf2c88c68b573b4900652ef6f0d407d7639f5e165790a7805b115f4bfd: Status 404 returned error can't find the container with id 517447bf2c88c68b573b4900652ef6f0d407d7639f5e165790a7805b115f4bfd
Oct 30 02:56:44.941896 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.941863    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:44.941993 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.941939    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:44.941993 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:44.941980    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:45.011617 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.011589    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.011666    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.011801    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.011843    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.011951    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.011974    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfa511b-667c-4f96-8509-785691b2cb17-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.011998    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqh8s\" (UniqueName: \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-kube-api-access-fqh8s\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.012055    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012458 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.012124    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfa511b-667c-4f96-8509-785691b2cb17-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012458 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.012163    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-certs\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012458 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.012185    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-data\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012458 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.012376    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-credential-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.012680 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.012612    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-socket\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.013256 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.013232    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfa511b-667c-4f96-8509-785691b2cb17-istiod-ca-cert\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.015204 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.015028    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfa511b-667c-4f96-8509-785691b2cb17-istio-podinfo\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.015204 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.015089    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-envoy\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.029271 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.029237    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-istio-token\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.029892 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.029868    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqh8s\" (UniqueName: \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-kube-api-access-fqh8s\") pod \"e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") " pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.166949 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.166789    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:45.225006 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.221337    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:45.225006 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.223418    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:56:45.225006 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.223473    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 02:56:45.749872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.746568    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" event={"ID":"17c5f62d-44a9-4d43-aff1-6f0191511107","Type":"ContainerStarted","Data":"b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93"}
Oct 30 02:56:45.792871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.787364    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" event={"ID":"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9","Type":"ContainerStarted","Data":"316cdf4f9842f88ac44c64d13befdf913df0f6172900c3f20202c3aca85be10e"}
Oct 30 02:56:45.792871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.787434    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" event={"ID":"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9","Type":"ContainerStarted","Data":"e07e96177b7bcd11112a26d97b8f1c0f1b2d81f3e456e7fc1d6f2f23c394ee20"}
Oct 30 02:56:45.826871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.817401    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" event={"ID":"30f8a79d-092c-443e-904a-7de32cd1af90","Type":"ContainerStarted","Data":"cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4"}
Oct 30 02:56:45.826871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.817463    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" event={"ID":"30f8a79d-092c-443e-904a-7de32cd1af90","Type":"ContainerStarted","Data":"517447bf2c88c68b573b4900652ef6f0d407d7639f5e165790a7805b115f4bfd"}
Oct 30 02:56:45.826871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.825949    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podStartSLOduration=2.825930164 podStartE2EDuration="2.825930164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:45.806823491 +0000 UTC m=+951.109001856" watchObservedRunningTime="2026-10-30 02:56:45.825930164 +0000 UTC m=+951.128108532"
Oct 30 02:56:45.888062 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.883914    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"]
Oct 30 02:56:45.895457 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.895386    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" podStartSLOduration=1.895365886 podStartE2EDuration="1.895365886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:45.8592731 +0000 UTC m=+951.161451471" watchObservedRunningTime="2026-10-30 02:56:45.895365886 +0000 UTC m=+951.197544252"
Oct 30 02:56:45.953162 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:56:45.944454    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfa511b_667c_4f96_8509_785691b2cb17.slice/crio-72848d7d0174b0988162216b96d1d3bc2b462e040a34f09dca0b5003484f7540 WatchSource:0}: Error finding container 72848d7d0174b0988162216b96d1d3bc2b462e040a34f09dca0b5003484f7540: Status 404 returned error can't find the container with id 72848d7d0174b0988162216b96d1d3bc2b462e040a34f09dca0b5003484f7540
Oct 30 02:56:45.986040 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.984729    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:45.986040 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.984834    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:45.986040 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:45.984896    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:46.007649 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.007556    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" podStartSLOduration=2.007532805 podStartE2EDuration="2.007532805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:45.931813805 +0000 UTC m=+951.233992176" watchObservedRunningTime="2026-10-30 02:56:46.007532805 +0000 UTC m=+951.309711171"
Oct 30 02:56:46.156481 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.156444    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:46.230998 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.230961    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:56:46.231266 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.231236    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 02:56:46.262224 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.262197    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:46.280868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.262511    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:56:46.283069 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.283034    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 02:56:46.401904 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.391226    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:46.596881 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.592466    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:46.596881 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.592928    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 02:56:46.596881 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.592982    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 02:56:46.630046 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.629366    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:46.630729 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.630704    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 02:56:46.630944 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.630920    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 02:56:46.885207 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.885106    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:46.892547 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.892513    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" event={"ID":"5bfa511b-667c-4f96-8509-785691b2cb17","Type":"ContainerStarted","Data":"8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8"}
Oct 30 02:56:46.901618 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.901589    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" event={"ID":"5bfa511b-667c-4f96-8509-785691b2cb17","Type":"ContainerStarted","Data":"72848d7d0174b0988162216b96d1d3bc2b462e040a34f09dca0b5003484f7540"}
Oct 30 02:56:46.945738 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:46.945680    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podStartSLOduration=2.94564855 podStartE2EDuration="2.94564855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:46.944194581 +0000 UTC m=+952.246372947" watchObservedRunningTime="2026-10-30 02:56:46.94564855 +0000 UTC m=+952.247826915"
Oct 30 02:56:47.188326 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.186985    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:47.189591 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.189284    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 02:56:47.189591 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.189336    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 02:56:47.224872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.222838    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:56:47.224872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.222925    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 02:56:47.270103 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.270070    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:56:47.270362 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.270334    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 02:56:47.286658 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.286213    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:47.313478 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.313436    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:47.460931 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.453741    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:47.460931 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.454964    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:47.571870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.570954    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:47.602874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.602501    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 02:56:47.602874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.602564    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 02:56:47.603141 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.603003    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 02:56:47.603141 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.603040    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 02:56:47.614878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.613955    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:47.653874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.632884    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 02:56:47.653874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.633238    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 02:56:47.700930 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.700004    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:56:47.814901 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.804952    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:47.887915 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.885995    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:47.887915 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:47.887110    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:56:48.182576 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.181483    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 02:56:48.182576 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.181544    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 02:56:48.224783 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.224730    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:56:48.225427 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.225378    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 02:56:48.261874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.255148    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:48.267376 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.267302    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:56:48.267618 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.267546    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 02:56:48.327377 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.327338    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:48.328315 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.328294    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:56:48.594869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.592200    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:56:48.594869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.592590    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:48.594869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.594134    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:48.594869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.594192    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:48.703996 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.703940    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:56:48.705434 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.705401    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Oct 30 02:56:48.705631 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.705611    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 02:56:48.706009 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.705988    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 02:56:48.706379 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.706356    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:48.716867 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.707580    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:56:48.717315 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.717295    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:56:48.751872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.717446    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:56:48.752161 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.752138    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:48.884926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.859209    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 02:56:48.884926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.859285    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 02:56:48.884926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.861268    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 02:56:48.884926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.861328    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 02:56:48.884926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.861401    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:56:48.913144 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:48.913095    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:49.241919 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.216253    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 02:56:49.286260 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.286211    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 02:56:49.590515 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.574896    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:49.608809 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.608260    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:49.609054 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.609025    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:56:49.613581 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.610934    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:56:49.613581 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.610975    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 02:56:49.613581 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.611906    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:56:49.613581 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.611940    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 02:56:49.683871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.670746    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 02:56:49.683871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.670816    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 02:56:49.711729 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.711683    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:49.714543 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.714511    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:56:49.715816 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.715783    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 02:56:49.741867 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.719275    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 02:56:49.867961 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.852964    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 02:56:49.871821 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.871784    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 02:56:49.882443 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.882406    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:56:49.883339 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.883321    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:56:49.940035 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:49.939984    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:50.280942 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.262502    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:50.306074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.288526    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:56:50.306074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.288582    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 02:56:50.306074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.288656    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:50.306074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.289522    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 02:56:50.306074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.289562    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 02:56:50.306074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.289792    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:56:50.306074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.289821    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 02:56:50.547080 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.547043    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": context deadline exceeded"
Oct 30 02:56:50.590691 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.590652    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 02:56:50.590955 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.590928    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 02:56:50.601228 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.601198    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 02:56:50.601459 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.601432    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 02:56:50.611980 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.611931    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:50.612529 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.612494    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 02:56:50.633880 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.625903    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:50.723989 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.723938    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:50.807566 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.724751    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:50.827668 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.724790    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:50.827668 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.731934    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:50.893577 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.893511    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:50.946673 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:50.946544    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:51.184325 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.184227    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 02:56:51.184617 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.184584    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 02:56:51.229757 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.229721    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:56:51.240410 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.240370    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 02:56:51.287672 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.286882    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:56:51.287672 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.286938    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 02:56:51.307977 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.300637    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:51.590545 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.590496    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 02:56:51.591219 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.590823    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 02:56:51.598200 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.598171    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 02:56:51.598376 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.598352    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 02:56:51.616463 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.616423    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:51.616969 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.616932    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:51.627992 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.627954    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:51.821874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.819992    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:51.821874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.821444    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 02:56:51.840926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.840386    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:51.840926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.840701    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 02:56:51.907872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.906959    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:51.949974 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:51.949924    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:52.205155 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.188725    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 02:56:52.205155 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.188792    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 02:56:52.205155 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.203973    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:52.225041 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.225001    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:56:52.225227 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.225063    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 02:56:52.225712 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.225655    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:52.273873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.272660    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:56:52.273873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.272718    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 02:56:52.619301 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.619256    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:56:52.620627 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.620591    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:52.622101 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.622079    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 02:56:52.622277 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.622251    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 02:56:52.623161 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.623138    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 02:56:52.623307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.623288    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 02:56:52.629959 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.629914    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:52.853873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.827479    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:52.930933 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.839936    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:52.930933 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.846727    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:52.930933 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.846759    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:56:52.959877 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.944179    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:56:52.959877 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:52.959131    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:53.175097 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.175040    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 02:56:53.189048 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.188942    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 02:56:53.221375 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.221337    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 02:56:53.223149 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.223109    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 02:56:53.261756 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.261721    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 02:56:53.262024 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.261993    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 02:56:53.337134 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.337099    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:56:53.563640 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.562647    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:56:53.582521 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.582498    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:56:53.591642 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.591618    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:56:53.620246 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.620215    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 02:56:53.620802 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.620760    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 02:56:53.621471 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.621448    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Oct 30 02:56:53.621634 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.621612    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Oct 30 02:56:53.622915 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.622880    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:53.663982 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.663955    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:56:53.924073 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.921252    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:53.924073 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.921639    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 02:56:53.973872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.950596    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:53.974668 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:53.965791    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:54.007024 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.006971    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:56:54.223931 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.198626    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 02:56:54.224244 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.224210    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 02:56:54.549935 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.549902    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:54.568059 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.568021    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:54.570626 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.570604    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:54.588507 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.587920    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:54.588860 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.588830    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 02:56:54.589155 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.589139    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 02:56:54.612470 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.610169    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:56:54.612470 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.611032    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:56:54.612470 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.611677    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:56:54.612941 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:54.612920    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:56:55.057016 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:55.056965    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:55.060171 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:55.060130    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:55.066041 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:55.066003    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 02:56:55.167349 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:55.167312    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 02:56:55.167621 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:55.167588    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 02:56:55.244872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:55.243755    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:55.248866 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:55.245193    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:55.252864 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:55.249529    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 02:56:55.267081 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:55.266896    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:55.453705 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:56:55.453580    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:56:56.167404 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:56.167364    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 02:56:56.167987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:56.167432    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 02:56:56.711070 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:56.711033    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:57.185087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:57.185058    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:57.193057 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:57.193034    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:57.197144 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:57.197124    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 02:56:57.819531 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:57.819501    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:59.095200 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.095164    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"]
Oct 30 02:56:59.100805 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.100784    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.129182 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.129158    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"]
Oct 30 02:56:59.147262 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.147226    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.147380 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.147269    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.147427 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.147387    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.147427 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.147416    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71c8959-e21c-4439-9de9-eb48ea50d938-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.147586 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.147566    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.147651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.147600    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.147651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.147630    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.147718 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.147675    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlwzh\" (UniqueName: \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-kube-api-access-xlwzh\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.147754 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.147716    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.249238 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.249209    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.249238 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.249243    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71c8959-e21c-4439-9de9-eb48ea50d938-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.249437 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.249307    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.249437 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.249322    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.249437 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.249339    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.249616 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.249467    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwzh\" (UniqueName: \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-kube-api-access-xlwzh\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.249616 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.249534    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.249723 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.249649    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.249723 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.249671    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.249723 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.249684    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.249918 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.249830    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.250102 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.250079    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.250193 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.250151    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71c8959-e21c-4439-9de9-eb48ea50d938-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.250244 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.250203    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.251858 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.251833    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.252066 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.252042    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.270453 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.270428    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.273666 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.273642    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlwzh\" (UniqueName: \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-kube-api-access-xlwzh\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.414712 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.414631    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:56:59.491594 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.491548    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:59.733006 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.732974    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"]
Oct 30 02:56:59.735948 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:56:59.735921    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71c8959_e21c_4439_9de9_eb48ea50d938.slice/crio-e57d1da29e74f9e0caad6b43c8fe26a7f4fccf70090005ac7ed86a51a1c05eaa WatchSource:0}: Error finding container e57d1da29e74f9e0caad6b43c8fe26a7f4fccf70090005ac7ed86a51a1c05eaa: Status 404 returned error can't find the container with id e57d1da29e74f9e0caad6b43c8fe26a7f4fccf70090005ac7ed86a51a1c05eaa
Oct 30 02:56:59.738600 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.738568    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:59.738682 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.738631    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:56:59.738682 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:56:59.738658    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:57:00.350462 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:00.350417    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" event={"ID":"f71c8959-e21c-4439-9de9-eb48ea50d938","Type":"ContainerStarted","Data":"43b412aec8a030612c8b3b99ef7643527cf47449cec03fffe5da15d62b20ce06"}
Oct 30 02:57:00.350896 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:00.350523    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" event={"ID":"f71c8959-e21c-4439-9de9-eb48ea50d938","Type":"ContainerStarted","Data":"e57d1da29e74f9e0caad6b43c8fe26a7f4fccf70090005ac7ed86a51a1c05eaa"}
Oct 30 02:57:00.395700 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:00.395640    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" podStartSLOduration=1.3956234730000001 podStartE2EDuration="1.395623473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:00.393719242 +0000 UTC m=+965.695897608" watchObservedRunningTime="2026-10-30 02:57:00.395623473 +0000 UTC m=+965.697801838"
Oct 30 02:57:00.415351 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:00.415325    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:57:00.417118 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:00.417089    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:57:00.417214 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:00.417144    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 02:57:01.168046 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:01.168012    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:01.415315 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:01.415281    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:57:01.415706 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:01.415339    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 02:57:02.415032 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:02.414998    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:57:02.415205 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:02.415049    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 02:57:02.645807 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:02.645767    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:03.415098 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:03.415062    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:57:03.415279 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:03.415126    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 02:57:04.145090 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:04.145055    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:04.415554 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:04.415476    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:57:04.415698 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:04.415548    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 02:57:05.415547 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:05.415514    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:57:05.415976 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:05.415562    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 02:57:05.789708 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:05.789622    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:06.415649 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:06.415610    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 02:57:06.416037 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:06.415690    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 02:57:07.285063 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:07.285011    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:07.456258 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:07.456219    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:57:07.458676 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:07.458652    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:57:07.460898 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:07.460878    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 02:57:09.686015 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:09.685965    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.688478 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:09.688441    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.705083 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:09.705051    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.705951 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:09.705928    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.727033 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:09.727001    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.727780 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:09.727757    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 02:57:09.868639 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:09.867938    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.917988 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:09.917927    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:09.988033 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:09.987931    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:57:10.280925 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.279163    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.421624 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.421578    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.652870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.642985    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:10.652870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.643093    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.705729 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.694800    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.705729 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.695319    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:57:10.705729 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.698105    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.705729 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.698426    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:57:10.707216 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.707185    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.712194 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.712173    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:57:10.802184 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.802140    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:57:10.821505 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.815107    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:57:10.821505 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.818128    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:57:10.821505 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.818208    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:57:10.824812 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.824095    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.825623 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.825127    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:57:10.829548 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.829504    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:57:10.829657 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.829550    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:10.871231 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.871192    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 02:57:10.879046 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.879021    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:57:10.905627 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:10.905553    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:57:11.010008 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.009978    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:57:11.010319 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.010287    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:11.012189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.010648    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:57:11.016922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.012879    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:57:11.025446 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.024177    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:57:11.034113 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.034083    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:11.095348 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.042119    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:57:11.095643 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.095627    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:57:11.106960 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.106933    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:57:11.139856 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.139802    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:57:11.160924 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.157720    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:57:11.161989 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.161532    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:57:11.190807 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:11.190661    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:12.982417 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:12.982384    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:14.486660 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:14.486627    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:16.293241 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:16.293199    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:17.982062 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:17.982030    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:19.584744 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:19.584709    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:21.242287 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:21.242247    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:23.083909 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:23.083870    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:24.877792 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:24.877759    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:25.445530 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:57:25.445497    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:57:26.366018 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:26.365983    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:27.899368 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:27.899325    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:29.193248 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.193211    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"]
Oct 30 02:57:29.198527 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.198507    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.228812 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.228772    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"]
Oct 30 02:57:29.375748 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.375717    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.375748 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.375749    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.375961 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.375781    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgnpr\" (UniqueName: \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-kube-api-access-jgnpr\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.375961 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.375944    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.376028 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.375975    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.376028 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.376008    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.376098 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.376071    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.376165 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.376152    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.376277 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.376266    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.418001 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.417974    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:29.478123 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.478023    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.478123 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.478109    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.478988 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.478953    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.479152 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.479047    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-credential-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.479206 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.479166    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.479242 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.479229    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.479275 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.479262    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.479326 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.479309    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgnpr\" (UniqueName: \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-kube-api-access-jgnpr\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.479426 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.479413    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.479464 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.479440    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istiod-ca-cert\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.479464 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.479451    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.480593 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.480570    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-data\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.480722 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.480675    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-certs\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.482309 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.480883    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-socket\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.484445 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.484420    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-envoy\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.484554 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.484539    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-podinfo\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.497348 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.497327    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-token\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.497577 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.497555    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgnpr\" (UniqueName: \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-kube-api-access-jgnpr\") pod \"e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") " pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.514322 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.514300    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:29.837926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.837896    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"]
Oct 30 02:57:29.842033 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:57:29.842007    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1227d4b_32b5_4ec8_9037_0b830b4528c6.slice/crio-4555d7f792d5e9d680b45fe25ff22acf10273cdaccbd129b7ae246b5d49abfe2 WatchSource:0}: Error finding container 4555d7f792d5e9d680b45fe25ff22acf10273cdaccbd129b7ae246b5d49abfe2: Status 404 returned error can't find the container with id 4555d7f792d5e9d680b45fe25ff22acf10273cdaccbd129b7ae246b5d49abfe2
Oct 30 02:57:29.844912 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.844837    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:57:29.844980 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.844961    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:57:29.845020 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.844992    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:57:29.973728 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.973683    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" event={"ID":"c1227d4b-32b5-4ec8-9037-0b830b4528c6","Type":"ContainerStarted","Data":"ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c"}
Oct 30 02:57:29.973906 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:29.973758    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" event={"ID":"c1227d4b-32b5-4ec8-9037-0b830b4528c6","Type":"ContainerStarted","Data":"4555d7f792d5e9d680b45fe25ff22acf10273cdaccbd129b7ae246b5d49abfe2"}
Oct 30 02:57:30.515488 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:30.515452    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:30.516696 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:30.516676    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:57:30.516785 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:30.516720    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 02:57:31.341884 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:31.340932    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:31.514947 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:31.514915    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:57:31.515127 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:31.514974    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 02:57:32.514739 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:32.514708    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:57:32.515643 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:32.514765    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 02:57:33.514933 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:33.514896    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:57:33.515321 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:33.514954    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 02:57:34.515212 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:34.515177    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:57:34.515607 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:34.515227    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 02:57:34.605214 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:34.605160    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:35.514863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:35.514816    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:57:35.515039 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:35.514893    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 02:57:36.202535 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:36.202503    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:36.342987 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:36.342949    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:36.515517 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:36.515434    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 02:57:36.515517 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:36.515503    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 02:57:37.520100 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:37.520069    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:37.520537 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:37.520517    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:37.521237 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:37.521216    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 02:57:37.573614 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:37.573561    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" podStartSLOduration=8.573548988 podStartE2EDuration="8.573548988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:30.017336162 +0000 UTC m=+995.319514529" watchObservedRunningTime="2026-10-30 02:57:37.573548988 +0000 UTC m=+1002.875727352"
Oct 30 02:57:37.810878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:37.810832    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:40.144727 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:40.144686    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:42.042874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:42.041940    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.042874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:42.042311    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 02:57:42.046870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:42.043975    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.055871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:42.052022    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.131361 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:42.131316    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.165151 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:42.161944    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.165151 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:42.162140    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:42.165151 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:42.163331    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 02:57:42.165151 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:42.163607    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Oct 30 02:57:42.926983 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:42.926923    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:43.073873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.049256    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:43.074751 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.073931    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:43.100891 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.073973    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 02:57:43.100891 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.074014    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:43.100891 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.075402    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:57:43.100891 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.085447    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:57:43.100891 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.085484    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:57:43.100891 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.085514    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:57:43.191974 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.134952    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:57:43.191974 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.137148    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:57:43.191974 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.164189    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 02:57:43.236872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.192686    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:57:43.236872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.193091    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:57:43.236872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.193383    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:57:43.236872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.193439    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:57:43.236872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.218278    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:57:43.979873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:43.951949    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 02:57:44.022137 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.022106    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:57:44.085012 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.084972    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 02:57:44.087727 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.087694    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:57:44.099139 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.098124    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:44.099139 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.098362    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 02:57:44.106753 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.106730    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:57:44.139686 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.139644    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:44.207051 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.207009    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:44.234922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.207493    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:57:44.363895 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.261297    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:57:44.680439 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.680410    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:57:44.703423 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.703396    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:57:44.734360 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.734328    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:57:44.973227 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.973143    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:57:44.973450 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.973437    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:57:44.973562 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.973553    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:57:44.973759 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.973747    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:57:44.979831 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:44.979349    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:57:46.226884 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:46.226836    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:47.662040 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:47.662004    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:49.587936 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:49.587896    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:51.071191 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:51.071155    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:52.863497 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:52.863465    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:54.400617 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:54.400579    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:55.446609 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:57:55.446573    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:57:56.163428 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:56.163391    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:57.996873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:57.973463    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"]
Oct 30 02:57:58.014138 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:58.014097    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerName="istio-proxy" containerID="cri-o://ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b" gracePeriod=30
Oct 30 02:57:59.425478 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.425441    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:59.744280 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.744205    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"]
Oct 30 02:57:59.748348 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.748332    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.776023 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.775999    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"]
Oct 30 02:57:59.872315 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.872286    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.872471 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.872357    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2b9l\" (UniqueName: \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-kube-api-access-h2b9l\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.872471 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.872451    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.872609 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.872590    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.872697 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.872681    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.872742 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.872721    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.872788 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.872750    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.872788 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.872768    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.872900 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.872793    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fa761a-333a-4f3d-b5f3-0828a4550456-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.974255 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.974220    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.974425 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.974266    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.974425 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.974299    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.974425 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.974316    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.974425 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.974339    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fa761a-333a-4f3d-b5f3-0828a4550456-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.974425 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.974370    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.974700 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.974459    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2b9l\" (UniqueName: \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-kube-api-access-h2b9l\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.974700 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.974500    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.974700 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.974590    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.974871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.974753    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.975123 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.975094    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.975123 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.975116    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.975307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.975165    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.975307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.975287    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fa761a-333a-4f3d-b5f3-0828a4550456-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.976815 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.976791    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.977067 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.977050    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.991549 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.991528    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:57:59.992108 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:57:59.992085    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2b9l\" (UniqueName: \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-kube-api-access-h2b9l\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:58:00.063071 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:00.063044    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:58:00.379749 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:00.379672    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"]
Oct 30 02:58:00.383371 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:58:00.383340    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fa761a_333a_4f3d_b5f3_0828a4550456.slice/crio-4948be2b2e2d7ce8a016eca8efb3a8af87d6976e732ac2db8d34de6c1a243df4 WatchSource:0}: Error finding container 4948be2b2e2d7ce8a016eca8efb3a8af87d6976e732ac2db8d34de6c1a243df4: Status 404 returned error can't find the container with id 4948be2b2e2d7ce8a016eca8efb3a8af87d6976e732ac2db8d34de6c1a243df4
Oct 30 02:58:00.385637 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:00.385609    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:58:00.385726 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:00.385671    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:58:00.385726 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:00.385701    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:58:00.933150 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:00.933120    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" event={"ID":"d2fa761a-333a-4f3d-b5f3-0828a4550456","Type":"ContainerStarted","Data":"fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a"}
Oct 30 02:58:00.933574 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:00.933172    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" event={"ID":"d2fa761a-333a-4f3d-b5f3-0828a4550456","Type":"ContainerStarted","Data":"4948be2b2e2d7ce8a016eca8efb3a8af87d6976e732ac2db8d34de6c1a243df4"}
Oct 30 02:58:00.982745 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:00.982703    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" podStartSLOduration=1.982688405 podStartE2EDuration="1.982688405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:00.980389733 +0000 UTC m=+1026.282568099" watchObservedRunningTime="2026-10-30 02:58:00.982688405 +0000 UTC m=+1026.284866799"
Oct 30 02:58:01.063669 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:01.063641    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:58:01.065496 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:01.065466    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:58:01.065602 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:01.065508    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 02:58:01.089712 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:01.089680    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:02.064097 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.064056    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:58:02.064585 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.064136    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 02:58:02.406791 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.406692    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"]
Oct 30 02:58:02.411427 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.411401    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.447289 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.447258    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d1788\"/\"distinct-w4-7d1788-kserve-self-signed-certs\""
Oct 30 02:58:02.447617 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.447599    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7d1788\"/\"openshift-service-ca.crt\""
Oct 30 02:58:02.447904 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.447840    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d1788\"/\"default-dockercfg-4qgt4\""
Oct 30 02:58:02.448657 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.448636    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7d1788\"/\"kube-root-ca.crt\""
Oct 30 02:58:02.453837 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.453810    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"]
Oct 30 02:58:02.490295 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.490269    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:02.508694 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.508669    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9zjd\" (UniqueName: \"kubernetes.io/projected/1318fb59-a096-440a-b3cb-43139a3bc387-kube-api-access-b9zjd\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.508814 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.508737    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-home\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.508814 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.508796    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-dshm\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.509035 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.509016    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1318fb59-a096-440a-b3cb-43139a3bc387-tls-certs\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.509092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.509079    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-model-cache\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.509194 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.509175    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-tmp-dir\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.610636 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.610598    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1318fb59-a096-440a-b3cb-43139a3bc387-tls-certs\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.611486 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.611456    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-model-cache\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.611617 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.611527    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-tmp-dir\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.611697 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.611620    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9zjd\" (UniqueName: \"kubernetes.io/projected/1318fb59-a096-440a-b3cb-43139a3bc387-kube-api-access-b9zjd\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.611697 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.611677    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-home\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.611800 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.611702    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-dshm\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.612020 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.611985    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-model-cache\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.612498 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.612457    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-tmp-dir\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.612608 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.612558    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-home\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.615621 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.615590    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-dshm\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.644567 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.644493    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1318fb59-a096-440a-b3cb-43139a3bc387-tls-certs\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.667322 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.667244    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9zjd\" (UniqueName: \"kubernetes.io/projected/1318fb59-a096-440a-b3cb-43139a3bc387-kube-api-access-b9zjd\") pod \"distinct-w4-7d1788-kserve-67df45cb4b-n2ch5\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") " pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.757484 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.757450    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:02.797317 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:02.797281    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:03.075427 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.075387    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:58:03.075812 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.075497    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 02:58:03.079564 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.079443    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"]
Oct 30 02:58:03.101983 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:58:03.101953    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1318fb59_a096_440a_b3cb_43139a3bc387.slice/crio-a581535c5e8453701353d4d60075473b57504ea94cfb84df9c099f4312389529 WatchSource:0}: Error finding container a581535c5e8453701353d4d60075473b57504ea94cfb84df9c099f4312389529: Status 404 returned error can't find the container with id a581535c5e8453701353d4d60075473b57504ea94cfb84df9c099f4312389529
Oct 30 02:58:03.362104 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.362073    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:58:03.447755 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.447716    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-socket\") pod \"bb2c275e-bd60-49f4-aa5a-1acca052b423\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") "
Oct 30 02:58:03.448394 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.448368    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-kube-api-access-vvmdn\" (UniqueName: \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-kube-api-access-vvmdn\") pod \"bb2c275e-bd60-49f4-aa5a-1acca052b423\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") "
Oct 30 02:58:03.449002 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.448977    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-podinfo\") pod \"bb2c275e-bd60-49f4-aa5a-1acca052b423\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") "
Oct 30 02:58:03.449103 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.448246    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-socket" pod "bb2c275e-bd60-49f4-aa5a-1acca052b423" (UID: "bb2c275e-bd60-49f4-aa5a-1acca052b423"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:58:03.449226 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.449211    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-envoy\") pod \"bb2c275e-bd60-49f4-aa5a-1acca052b423\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") "
Oct 30 02:58:03.449289 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.449242    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2c275e-bd60-49f4-aa5a-1acca052b423-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c275e-bd60-49f4-aa5a-1acca052b423-istiod-ca-cert\") pod \"bb2c275e-bd60-49f4-aa5a-1acca052b423\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") "
Oct 30 02:58:03.449339 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.449286    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-token\") pod \"bb2c275e-bd60-49f4-aa5a-1acca052b423\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") "
Oct 30 02:58:03.449394 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.449356    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-credential-socket\") pod \"bb2c275e-bd60-49f4-aa5a-1acca052b423\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") "
Oct 30 02:58:03.449457 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.449442    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-data\") pod \"bb2c275e-bd60-49f4-aa5a-1acca052b423\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") "
Oct 30 02:58:03.449508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.449495    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-certs\") pod \"bb2c275e-bd60-49f4-aa5a-1acca052b423\" (UID: \"bb2c275e-bd60-49f4-aa5a-1acca052b423\") "
Oct 30 02:58:03.450146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.450046    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.450262 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.450230    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-certs" pod "bb2c275e-bd60-49f4-aa5a-1acca052b423" (UID: "bb2c275e-bd60-49f4-aa5a-1acca052b423"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:58:03.450397 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.450367    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-data" pod "bb2c275e-bd60-49f4-aa5a-1acca052b423" (UID: "bb2c275e-bd60-49f4-aa5a-1acca052b423"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:58:03.452509 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.452487    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-kube-api-access-vvmdn" pod "bb2c275e-bd60-49f4-aa5a-1acca052b423" (UID: "bb2c275e-bd60-49f4-aa5a-1acca052b423"). InnerVolumeSpecName "kube-api-access-vvmdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:03.453544 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.453520    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-credential-socket" pod "bb2c275e-bd60-49f4-aa5a-1acca052b423" (UID: "bb2c275e-bd60-49f4-aa5a-1acca052b423"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:58:03.453909 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.453884    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2c275e-bd60-49f4-aa5a-1acca052b423-istiod-ca-cert" pod "bb2c275e-bd60-49f4-aa5a-1acca052b423" (UID: "bb2c275e-bd60-49f4-aa5a-1acca052b423"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:03.454059 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.454039    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-token" pod "bb2c275e-bd60-49f4-aa5a-1acca052b423" (UID: "bb2c275e-bd60-49f4-aa5a-1acca052b423"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:03.454254 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.454228    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-podinfo" pod "bb2c275e-bd60-49f4-aa5a-1acca052b423" (UID: "bb2c275e-bd60-49f4-aa5a-1acca052b423"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 02:58:03.455431 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.455407    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-envoy" pod "bb2c275e-bd60-49f4-aa5a-1acca052b423" (UID: "bb2c275e-bd60-49f4-aa5a-1acca052b423"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:58:03.551960 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.551922    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.551960 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.551958    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2c275e-bd60-49f4-aa5a-1acca052b423-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.551960 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.551968    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.552205 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.551976    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.552205 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.551986    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.552205 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.551994    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2c275e-bd60-49f4-aa5a-1acca052b423-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.552205 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.552003    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmdn\" (UniqueName: \"kubernetes.io/projected/bb2c275e-bd60-49f4-aa5a-1acca052b423-kube-api-access-vvmdn\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.552205 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.552011    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2c275e-bd60-49f4-aa5a-1acca052b423-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 02:58:03.971582 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.971476    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" event={"ID":"1318fb59-a096-440a-b3cb-43139a3bc387","Type":"ContainerStarted","Data":"733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb"}
Oct 30 02:58:03.971582 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.971524    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" event={"ID":"1318fb59-a096-440a-b3cb-43139a3bc387","Type":"ContainerStarted","Data":"a581535c5e8453701353d4d60075473b57504ea94cfb84df9c099f4312389529"}
Oct 30 02:58:03.972961 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.972933    5061 generic.go:356] "Generic (PLEG): container finished" podID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerID="ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b" exitCode=0
Oct 30 02:58:03.973093 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.972982    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" event={"ID":"bb2c275e-bd60-49f4-aa5a-1acca052b423","Type":"ContainerDied","Data":"ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b"}
Oct 30 02:58:03.973093 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.973012    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp" event={"ID":"bb2c275e-bd60-49f4-aa5a-1acca052b423","Type":"ContainerDied","Data":"a51c83efa0b67a04abd0be3ca4f00b8e11fe3a6713d38a7badf9611c0e8614e4"}
Oct 30 02:58:03.973093 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.972980    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"
Oct 30 02:58:03.973259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.973031    5061 scope.go:122] "RemoveContainer" containerID="ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b"
Oct 30 02:58:03.983457 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.983440    5061 scope.go:122] "RemoveContainer" containerID="ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b"
Oct 30 02:58:03.983711 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:58:03.983688    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b\": container with ID starting with ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b not found: ID does not exist" containerID="ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b"
Oct 30 02:58:03.983769 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:03.983712    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b"} err="failed to get container status \"ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b\": rpc error: code = NotFound desc = could not find container \"ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b\": container with ID starting with ab9459665135f39651a9877d0571e4b2155c49309cf3d3a52c9c87452ef36e2b not found: ID does not exist"
Oct 30 02:58:04.008462 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:04.008415    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podStartSLOduration=2.008402264 podStartE2EDuration="2.008402264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:04.006964932 +0000 UTC m=+1029.309143279" watchObservedRunningTime="2026-10-30 02:58:04.008402264 +0000 UTC m=+1029.310580629"
Oct 30 02:58:04.064003 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:04.063972    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:58:04.064189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:04.064028    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 02:58:04.064441 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:04.064417    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"]
Oct 30 02:58:04.074080 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:04.074055    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hc2qp"]
Oct 30 02:58:04.758282 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:04.758249    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:04.851294 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:04.851265    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" path="/var/lib/kubelet/pods/bb2c275e-bd60-49f4-aa5a-1acca052b423/volumes"
Oct 30 02:58:04.943901 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:04.939251    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:05.063801 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:05.063765    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:58:05.064057 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:05.064035    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 02:58:06.064423 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:06.064387    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:58:06.064830 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:06.064440    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 02:58:07.063720 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:07.063682    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 02:58:07.063916 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:07.063739    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 02:58:07.539588 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:07.539496    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:08.068032 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:08.068001    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:58:08.068396 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:08.068376    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:58:08.069012 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:08.068990    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 02:58:08.075111 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:08.075085    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:10.070716 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:10.070684    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:11.703146 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:11.703107    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:12.762863 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:12.762804    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:12.775393 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:12.775369    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:13.063312 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.063282    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:13.154488 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.154455    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"]
Oct 30 02:58:13.155102 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.155080    5061 cpu_manager.go:414] "containerMap: removing container" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerName="istio-proxy"
Oct 30 02:58:13.155102 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.155099    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerName="istio-proxy"
Oct 30 02:58:13.155320 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.155232    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2c275e-bd60-49f4-aa5a-1acca052b423" containerName="istio-proxy"
Oct 30 02:58:13.158787 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.158768    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.175214 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.175187    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"distinct-w6-32bad3-kserve-self-signed-certs\""
Oct 30 02:58:13.175342 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.175301    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"default-dockercfg-g2vqc\""
Oct 30 02:58:13.175417 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.175337    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"openshift-service-ca.crt\""
Oct 30 02:58:13.179481 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.179441    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"]
Oct 30 02:58:13.182278 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.182253    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-home\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.182531 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.182504    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-tmp-dir\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.182628 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.182612    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcxg\" (UniqueName: \"kubernetes.io/projected/459a2b31-dffd-42f3-bf06-907d543e23e0-kube-api-access-kwcxg\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.182696 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.182673    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-model-cache\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.182754 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.182718    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459a2b31-dffd-42f3-bf06-907d543e23e0-tls-certs\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.182812 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.182790    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-dshm\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.184835 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.184818    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-32bad3\"/\"kube-root-ca.crt\""
Oct 30 02:58:13.221232 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.221188    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:13.284238 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.284212    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459a2b31-dffd-42f3-bf06-907d543e23e0-tls-certs\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.284386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.284270    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-dshm\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.284386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.284308    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-home\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.284386 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.284343    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-tmp-dir\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.284538 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.284391    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcxg\" (UniqueName: \"kubernetes.io/projected/459a2b31-dffd-42f3-bf06-907d543e23e0-kube-api-access-kwcxg\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.284538 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.284427    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-model-cache\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.284767 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.284742    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-home\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.284879 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.284794    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-tmp-dir\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.284879 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.284870    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-model-cache\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.286542 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.286516    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-dshm\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.286888 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.286843    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459a2b31-dffd-42f3-bf06-907d543e23e0-tls-certs\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.303693 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.303670    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcxg\" (UniqueName: \"kubernetes.io/projected/459a2b31-dffd-42f3-bf06-907d543e23e0-kube-api-access-kwcxg\") pod \"distinct-w6-32bad3-kserve-64b6968fbb-qqwvf\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") " pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.474078 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.473987    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:13.514521 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.514485    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:13.777634 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:13.777557    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"]
Oct 30 02:58:13.780896 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:58:13.780864    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod459a2b31_dffd_42f3_bf06_907d543e23e0.slice/crio-b6ebda5bc828b0a4b2df0d608e61c52260d33962e33b52c5b7eb258de128a1c8 WatchSource:0}: Error finding container b6ebda5bc828b0a4b2df0d608e61c52260d33962e33b52c5b7eb258de128a1c8: Status 404 returned error can't find the container with id b6ebda5bc828b0a4b2df0d608e61c52260d33962e33b52c5b7eb258de128a1c8
Oct 30 02:58:14.061195 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:14.061163    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" event={"ID":"459a2b31-dffd-42f3-bf06-907d543e23e0","Type":"ContainerStarted","Data":"38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152"}
Oct 30 02:58:14.061371 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:14.061211    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" event={"ID":"459a2b31-dffd-42f3-bf06-907d543e23e0","Type":"ContainerStarted","Data":"b6ebda5bc828b0a4b2df0d608e61c52260d33962e33b52c5b7eb258de128a1c8"}
Oct 30 02:58:14.097956 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:14.097272    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podStartSLOduration=1.097256287 podStartE2EDuration="1.097256287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:14.096814756 +0000 UTC m=+1039.398993122" watchObservedRunningTime="2026-10-30 02:58:14.097256287 +0000 UTC m=+1039.399434654"
Oct 30 02:58:14.474986 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:14.474912    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:14.922515 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:14.922478    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"]
Oct 30 02:58:14.926689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:14.926666    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:14.948801 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:14.948770    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"]
Oct 30 02:58:15.010802 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.010773    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.010978 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.010811    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.010978 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.010881    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.010978 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.010959    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.011087 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.010987    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.011125 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.011104    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.011156 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.011130    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.011197 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.011182    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.011291 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.011274    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6fjs\" (UniqueName: \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-kube-api-access-z6fjs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.112463 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.112433    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.112651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.112478    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.112651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.112511    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.112651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.112540    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.112651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.112557    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.112651 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.112647    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.112981 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.112677    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.112981 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.112737    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.112981 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.112817    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6fjs\" (UniqueName: \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-kube-api-access-z6fjs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.112981 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.112893    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.112981 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.112909    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.113259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.113176    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.113259 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.113224    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.113370 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.113351    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.115150 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.115116    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.115408 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.115391    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.130051 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.130025    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.130266 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.130245    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6fjs\" (UniqueName: \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-kube-api-access-z6fjs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.241074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.240988    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:15.301214 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.301155    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:15.562258 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.562233    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"]
Oct 30 02:58:15.565110 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:58:15.565083    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7621da70_c0e2_4d2b_983d_1cf05bb7e844.slice/crio-d44b46987c62cf7920e9b8497b34d17480670cbc69c954e3357f36d77c6ae687 WatchSource:0}: Error finding container d44b46987c62cf7920e9b8497b34d17480670cbc69c954e3357f36d77c6ae687: Status 404 returned error can't find the container with id d44b46987c62cf7920e9b8497b34d17480670cbc69c954e3357f36d77c6ae687
Oct 30 02:58:15.567917 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.567884    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:58:15.568001 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.567949    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:58:15.568001 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:15.567975    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:58:16.075778 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:16.075741    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" event={"ID":"7621da70-c0e2-4d2b-983d-1cf05bb7e844","Type":"ContainerStarted","Data":"80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7"}
Oct 30 02:58:16.076203 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:16.075803    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" event={"ID":"7621da70-c0e2-4d2b-983d-1cf05bb7e844","Type":"ContainerStarted","Data":"d44b46987c62cf7920e9b8497b34d17480670cbc69c954e3357f36d77c6ae687"}
Oct 30 02:58:16.135448 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:16.135401    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podStartSLOduration=2.135387399 podStartE2EDuration="2.135387399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:16.123727009 +0000 UTC m=+1041.425905373" watchObservedRunningTime="2026-10-30 02:58:16.135387399 +0000 UTC m=+1041.437565764"
Oct 30 02:58:16.241539 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:16.241507    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:16.243326 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:16.243299    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:58:16.243469 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:16.243348    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 02:58:17.175333 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:17.175301    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:17.247581 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:17.244285    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:58:17.247581 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:17.244347    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 02:58:18.244234 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:18.244191    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:58:18.244658 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:18.244257    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 02:58:18.791053 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:18.791013    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:19.244946 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:19.244902    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:58:19.245528 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:19.245495    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 02:58:20.252869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:20.241558    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:58:20.252869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:20.241635    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 02:58:21.109883 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:21.108929    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:21.113880 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:21.110381    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:58:21.120003 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:21.119971    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:21.120508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:21.120477    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:21.243271 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:21.243233    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:58:21.243553 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:21.243520    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 02:58:21.268971 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:21.268925    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:21.968378 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:21.968334    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:21.987873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:21.985921    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.020047 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.019998    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:22.020644 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.020611    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.067493 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.067436    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.091702 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.091167    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.113180 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.113137    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.162874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.116907    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 02:58:22.162874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.117654    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:58:22.162874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.117714    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:58:22.188872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.163315    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 02:58:22.227926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.195264    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 02:58:22.227926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.197335    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:58:22.227926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.197388    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:58:22.284903 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.252039    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Oct 30 02:58:22.346936 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.286688    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:58:22.346936 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.286743    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 02:58:22.346936 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.286818    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.346936 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.288403    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:58:22.733874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.708501    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:22.968870 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.968811    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:22.993973 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:22.992312    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.068631 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.068574    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 02:58:23.083361 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.083321    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:23.083637 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.083603    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.098871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.095017    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 02:58:23.109971 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.109940    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:58:23.110217 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.110201    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:58:23.110361 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.110348    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:23.110491 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.110480    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:58:23.126748 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.126709    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:58:23.138911 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.126975    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.186873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.165713    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.212600 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.195936    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.290736 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.290632    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.342724 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.342687    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:58:23.343018 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.342987    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 02:58:23.600869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.562902    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:23.633097 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.631984    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:23.633097 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.632050    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.894928 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.889280    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.960872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.934962    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:23.960872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.935344    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:58:23.984009 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:23.983958    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.022970 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.022920    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:58:24.125874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.122671    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.241932 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.166003    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 02:58:24.241932 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.167296    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.241932 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.168600    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:58:24.241932 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.230989    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.241932 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.231395    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:24.265463 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.258710    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:58:24.265463 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.258770    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 02:58:24.350940 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.334482    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:58:24.602880 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.595037    5061 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Oct 30 02:58:24.958342 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.958062    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 02:58:24.991196 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:24.991133    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.057121 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.057058    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.137903 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.137735    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.151690 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.151552    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 02:58:25.168687 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.168638    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.169028 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.168990    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.169306 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.169280    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.327313 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.327272    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.330043 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.330012    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.365029 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.364998    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 02:58:25.365625 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.365580    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 02:58:25.369615 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.369566    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:25.413054 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.413028    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:58:25.434883 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.430327    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:58:25.485021 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.437894    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:58:25.485021 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.461490    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:25.577528 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.576521    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:58:25.602038 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:58:25.601931    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:58:25.604220 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.604198    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:25.610752 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.610731    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:58:25.646177 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.646153    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:58:25.656938 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.656915    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:58:25.667558 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.667539    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:58:25.683864 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:25.683822    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:58:26.926929 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:26.926897    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:26.927829 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:26.927802    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:26.928508 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:26.928491    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 02:58:29.451065 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:29.451026    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:29.649733 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:29.649697    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:31.150421 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:31.150383    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:32.619520 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:32.619484    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:33.485597 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:33.485568    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:33.496573 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:33.496548    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:34.198084 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:34.198051    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:35.786397 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:35.786361    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:37.347690 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:37.347659    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:38.876058 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:38.876028    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:40.519839 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:40.519806    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:42.087841 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:42.087802    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:43.632073 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:43.632040    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:45.086390 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.086358    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:45.669065 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.669033    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"]
Oct 30 02:58:45.678435 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.678401    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.698011 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.697984    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 02:58:45.698153 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.698058    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 02:58:45.698153 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.698019    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 02:58:45.705205 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.705165    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ctkqx\""
Oct 30 02:58:45.709509 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.709488    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"]
Oct 30 02:58:45.757426 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.757403    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.757567 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.757435    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdtd\" (UniqueName: \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-kube-api-access-4jdtd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.757642 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.757595    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.757694 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.757645    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.757799 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.757778    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.757884 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.757841    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.757938 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.757893    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.859312 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.859288    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.859312 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.859321    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.859535 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.859395    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.859535 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.859425    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdtd\" (UniqueName: \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-kube-api-access-4jdtd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.859648 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.859547    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.859648 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.859584    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.859753 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.859732    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.860445 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.860414    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.861986 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.861955    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.862075 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.861968    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.862122 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.862095    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.862157 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.862130    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.878790 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.878768    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdtd\" (UniqueName: \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-kube-api-access-4jdtd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.882191 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.882171    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5fnmn\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:45.992189 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:45.992118    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:46.310162 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:46.310135    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"]
Oct 30 02:58:46.312920 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:58:46.312893    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5f28eb_0c2b_4f73_a1bb_7977cdb5abbb.slice/crio-2fd7c37c50b39b747ff9065d54b5aa5d9327d2e0fe1f55b2e94fc59bbeb4e29c WatchSource:0}: Error finding container 2fd7c37c50b39b747ff9065d54b5aa5d9327d2e0fe1f55b2e94fc59bbeb4e29c: Status 404 returned error can't find the container with id 2fd7c37c50b39b747ff9065d54b5aa5d9327d2e0fe1f55b2e94fc59bbeb4e29c
Oct 30 02:58:46.315689 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:46.315673    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:58:46.352814 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:46.352745    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:46.521624 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:46.521591    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn" event={"ID":"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb","Type":"ContainerStarted","Data":"2fd7c37c50b39b747ff9065d54b5aa5d9327d2e0fe1f55b2e94fc59bbeb4e29c"}
Oct 30 02:58:46.717226 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:46.717136    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:49.128069 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:49.128027    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.142708 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:49.141943    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.186030 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:49.184951    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.217220 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:49.217168    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.337004 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:49.336950    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.464929 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:49.463993    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:58:49.488881 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:49.487366    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:49.981872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:49.973976    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.005872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.001023    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.034875 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.032930    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.183872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.179557    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.183872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.181742    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.183872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.183141    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:58:50.199872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.199259    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:58:50.199872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.199536    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:58:50.224871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.200361    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.224871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.200584    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:58:50.228868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.225942    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.228868 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.226277    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:58:50.285925 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.285685    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.350919 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.337282    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 02:58:50.350919 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.338603    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:58:50.508892 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.436380    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.508892 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.472935    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.508892 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.473283    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:58:50.508892 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.490419    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:50.508892 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.490708    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:50.596264 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.596222    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:50.596553 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:50.596517    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.081532 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:51.081478    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.285616 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:51.285563    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.368923 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:51.361634    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:51.491875 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:51.469064    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.889080 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:51.496089    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.891564 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:51.531136    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:51.892452 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:51.531500    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:58:51.892599 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:51.531521    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 02:58:51.896083 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:51.531536    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 02:58:51.906310 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:51.535897    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:58:51.916537 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:51.537293    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.096055 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.096009    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 02:58:52.166896 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.166785    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:58:52.177066 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.176990    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:58:52.242146 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:58:52.242114    5061 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.371s"
Oct 30 02:58:52.270649 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.270621    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:58:52.271299 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.271279    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:58:52.272562 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.272547    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:58:52.273598 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.273573    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:52.295599 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.295577    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:58:52.296941 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.286378    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 02:58:52.612583 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.612370    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Oct 30 02:58:52.702476 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.702437    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:58:52.734463 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.734429    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:58:52.737203 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.737116    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:58:52.738589 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.738566    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:58:52.816991 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.816964    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:58:52.835159 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.818694    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:58:52.840123 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.840027    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:58:52.911922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.906672    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:58:52.911922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.906730    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:58:52.911922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.906762    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:58:52.911922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:52.906793    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:58:54.478422 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:54.478385    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:58:54.478810 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:54.478466    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:58:55.138657 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:55.138616    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn" event={"ID":"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb","Type":"ContainerStarted","Data":"2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783"}
Oct 30 02:58:55.138955 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:55.138891    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:55.141303 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:55.141273    5061 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5fnmn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 02:58:55.141414 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:55.141334    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn" podUID="2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 02:58:55.278502 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:55.278465    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:55.447412 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:58:55.447375    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:58:55.452121 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:55.452095    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:56.145250 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:56.145224    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 02:58:56.195165 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:56.195117    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn" podStartSLOduration=3.033719591 podStartE2EDuration="11.195099738s" totalImagesPullingTime="8.161380147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:55.204014631 +0000 UTC m=+1080.506192993" watchObservedRunningTime="2026-10-30 02:58:56.195099738 +0000 UTC m=+1081.497278103"
Oct 30 02:58:58.043250 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:58.043215    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:59.715355 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:58:59.715325    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:01.492227 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:01.492196    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:03.103319 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:03.103285    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:04.863671 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:04.863631    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:06.485522 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:06.485492    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:07.975924 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:07.975891    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:09.783344 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:09.783308    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:11.412593 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:11.412558    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:13.089320 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:13.089288    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:14.579090 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:14.579058    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:15.165590 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.165554    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"]
Oct 30 02:59:15.169687 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.169662    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.192326 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.192295    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"]
Oct 30 02:59:15.241999 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.241968    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcsf\" (UniqueName: \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-kube-api-access-4dcsf\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.241999 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.242003    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.242251 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.242032    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.242251 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.242149    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.242370 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.242285    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.242370 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.242326    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.242370 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.242348    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.242534 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.242379    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.242534 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.242500    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.343698 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.343661    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.343698 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.343702    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.343979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.343720    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.343979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.343755    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.343979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.343782    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.343979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.343865    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcsf\" (UniqueName: \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-kube-api-access-4dcsf\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.343979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.343896    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.343979 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.343944    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.344251 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.344025    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.344251 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.344179    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-credential-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.344362 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.344335    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-data\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.344409 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.344381    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-socket\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.344533 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.344509    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-certs\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.344669 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.344611    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istiod-ca-cert\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.346580 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.346552    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-envoy\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.346683 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.346643    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-podinfo\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.362600 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.362570    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-token\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.362713 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.362697    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcsf\" (UniqueName: \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-kube-api-access-4dcsf\") pod \"e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") " pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.488978 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.488899    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:15.808682 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.808655    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"]
Oct 30 02:59:15.812435 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:59:15.812406    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eec092_64ae_43a2_8c3a_7c1e0f27b168.slice/crio-4d457c3f2c07bc4b2dea3be510613e73bdb2f74087165f62d62f7ce884da2bf4 WatchSource:0}: Error finding container 4d457c3f2c07bc4b2dea3be510613e73bdb2f74087165f62d62f7ce884da2bf4: Status 404 returned error can't find the container with id 4d457c3f2c07bc4b2dea3be510613e73bdb2f74087165f62d62f7ce884da2bf4
Oct 30 02:59:15.815472 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.815438    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:59:15.815580 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.815549    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:59:15.815640 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:15.815598    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:59:16.282750 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:16.282662    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" event={"ID":"f8eec092-64ae-43a2-8c3a-7c1e0f27b168","Type":"ContainerStarted","Data":"ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6"}
Oct 30 02:59:16.282750 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:16.282743    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" event={"ID":"f8eec092-64ae-43a2-8c3a-7c1e0f27b168","Type":"ContainerStarted","Data":"4d457c3f2c07bc4b2dea3be510613e73bdb2f74087165f62d62f7ce884da2bf4"}
Oct 30 02:59:16.337252 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:16.337185    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podStartSLOduration=1.337166716 podStartE2EDuration="1.337166716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:16.331934821 +0000 UTC m=+1101.634113185" watchObservedRunningTime="2026-10-30 02:59:16.337166716 +0000 UTC m=+1101.639345082"
Oct 30 02:59:16.454939 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:16.453983    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:16.490078 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:16.490043    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:16.491922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:16.491895    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:16.492069 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:16.491953    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:17.492874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:17.491187    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:17.492874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:17.491251    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:18.491879 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:18.491284    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:18.491879 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:18.491361    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:18.844873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:18.844325    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.870873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:18.870053    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.874869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:18.872496    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:59:18.895872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:18.875923    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.895872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:18.876637    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.895872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:18.876949    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 02:59:18.895872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:18.890388    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.895872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:18.894916    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:18.923873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:18.923256    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 02:59:19.318873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.295968    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.491878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.490872    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:19.491878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.491039    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:19.756927 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.750389    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.853874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.852393    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.853874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.852931    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:59:19.889873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.887945    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.889873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.889304    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:59:19.899871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.894015    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 02:59:19.899871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.895252    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:59:19.899871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.895315    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:59:19.899871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.895352    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:59:19.911878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.906937    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.911878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.907262    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:59:19.911878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.908007    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.911878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.908202    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:59:19.911878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.908393    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:59:19.911878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.908441    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:59:19.920921 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.913333    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.925869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.921956    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:59:19.925869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.924937    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:19.930874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:19.926100    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:59:20.318875 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.318675    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:20.319065 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.318954    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 02:59:20.365307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.338278    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:59:20.475872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.474919    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": context deadline exceeded"
Oct 30 02:59:20.562248 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.557413    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:20.562248 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.557477    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:20.562248 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.560486    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:20.562248 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.560533    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:20.772726 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.761968    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:20.772726 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.763855    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:59:20.853109 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.853063    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:20.888471 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.888429    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 02:59:20.893122 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.893074    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.084933 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.916419    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.122000 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.916481    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.127935 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.916502    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 02:59:21.185997 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.916521    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:59:21.189913 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:20.916561    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 02:59:21.215988 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:21.048033    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.407553 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:21.407446    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 02:59:21.519899 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:21.519496    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:21.519899 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:21.519593    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:21.801941 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:21.792165    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:21.916920 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:59:21.845778    5061 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.004s"
Oct 30 02:59:21.976866 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:21.917506    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:21.977416 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:21.917584    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.074926 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.065753    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.122108 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.122047    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.165872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.135205    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.165872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.135249    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:59:22.189449 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.187343    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.189449 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.188545    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.197659 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.196951    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.220042 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.219930    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.408118 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.408014    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.492878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.490372    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:22.492878 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.490433    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:22.569046 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.569005    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:22.569383 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.569352    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.793928 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.793770    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:22.904871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.903904    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.979872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.978948    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:22.979872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:22.979364    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 02:59:23.067053 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.066951    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 02:59:23.103872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.098054    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.103872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.098138    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.168176 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.168127    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.170117 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.168430    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 02:59:23.170572 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.169981    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 02:59:23.189466 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.189412    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.190765 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.190075    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.199669 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.198929    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.410464 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.410362    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.452307 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.452265    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:23.452607 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.452573    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.489675 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.489636    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:23.489996 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.489953    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:23.801017 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.800960    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.909919 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.907946    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.985874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.985104    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:23.985874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:23.985501    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.070107 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.069992    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.213928 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.195775    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.236869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.221993    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.236869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.222399    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.264155 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.264105    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.288620 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.264149    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.309332 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.268091    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 02:59:24.425427 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.425323    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.481380 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.481337    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:24.491477 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.491446    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:24.502353 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.498912    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:24.782988 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.782905    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:59:24.792235 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.792215    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:59:24.802116 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.802090    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:59:24.893444 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.893411    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:59:24.893740 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.893484    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:59:24.893740 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.893515    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:59:24.946255 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.946228    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:59:24.956082 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.955044    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:59:24.956885 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.956859    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:59:24.957726 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.957709    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:59:24.975389 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:24.975360    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:59:25.478396 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:59:25.478360    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:59:25.489815 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:25.489789    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:25.489937 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:25.489843    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:26.489247 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:26.489207    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:26.489717 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:26.489265    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:27.489641 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:27.489609    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 30 02:59:27.490030 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:27.489657    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 30 02:59:27.980258 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:27.980217    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:28.610676 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:28.610620    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:28.770026 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:28.769965    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:28.915499 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:28.915422    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:29.405014 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:29.404980    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:29.405378 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:29.405359    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:29.406278 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:29.406258    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 02:59:30.383695 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:30.383664    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:30.966317 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:30.966286    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:31.435323 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:31.435294    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:31.567610 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:31.567577    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:31.705090 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:31.705011    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:33.743523 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:33.743490    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:35.937204 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:35.937116    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:36.380276 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:36.380241    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:36.529380 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:36.529338    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:36.718187 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:36.718110    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:37.193064 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:37.193026    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:37.342831 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:37.342789    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:37.507176 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:37.507102    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:38.666791 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:38.666756    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:38.921304 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:38.921210    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:39.065871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:39.060452    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:40.676141 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:40.676110    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:40.805084 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:40.805049    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:40.820679 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:40.820649    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"]
Oct 30 02:59:40.824506 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:40.824486    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 02:59:40.850161 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:40.850134    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"]
Oct 30 02:59:40.870132 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:40.870107    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-93fe9de7\""
Oct 30 02:59:40.870695 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:40.870147    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-93fe9de7-dockercfg-p5f5d\""
Oct 30 02:59:40.987928 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:40.958744    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzj49\" (UniqueName: \"kubernetes.io/projected/11a1f64d-e684-429a-899b-7f5d0b45612c-kube-api-access-vzj49\") pod \"payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45\" (UID: \"11a1f64d-e684-429a-899b-7f5d0b45612c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 02:59:40.987928 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:40.969091    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a1f64d-e684-429a-899b-7f5d0b45612c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45\" (UID: \"11a1f64d-e684-429a-899b-7f5d0b45612c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 02:59:41.070937 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:41.070889    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzj49\" (UniqueName: \"kubernetes.io/projected/11a1f64d-e684-429a-899b-7f5d0b45612c-kube-api-access-vzj49\") pod \"payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45\" (UID: \"11a1f64d-e684-429a-899b-7f5d0b45612c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 02:59:41.071124 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:41.071040    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a1f64d-e684-429a-899b-7f5d0b45612c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45\" (UID: \"11a1f64d-e684-429a-899b-7f5d0b45612c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 02:59:41.071902 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:41.071842    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a1f64d-e684-429a-899b-7f5d0b45612c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45\" (UID: \"11a1f64d-e684-429a-899b-7f5d0b45612c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 02:59:41.090588 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:41.090562    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzj49\" (UniqueName: \"kubernetes.io/projected/11a1f64d-e684-429a-899b-7f5d0b45612c-kube-api-access-vzj49\") pod \"payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45\" (UID: \"11a1f64d-e684-429a-899b-7f5d0b45612c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 02:59:41.139663 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:41.139631    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 02:59:41.423670 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:41.423639    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:41.465732 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:41.465696    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"]
Oct 30 02:59:41.470538 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:59:41.470507    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a1f64d_e684_429a_899b_7f5d0b45612c.slice/crio-eb5f472b8184db7eb35b08f89a03f3dc14dee8104f7471287a1f169bb41b7635 WatchSource:0}: Error finding container eb5f472b8184db7eb35b08f89a03f3dc14dee8104f7471287a1f169bb41b7635: Status 404 returned error can't find the container with id eb5f472b8184db7eb35b08f89a03f3dc14dee8104f7471287a1f169bb41b7635
Oct 30 02:59:41.621967 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:41.621933    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45" event={"ID":"11a1f64d-e684-429a-899b-7f5d0b45612c","Type":"ContainerStarted","Data":"87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a"}
Oct 30 02:59:41.622096 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:41.621992    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45" event={"ID":"11a1f64d-e684-429a-899b-7f5d0b45612c","Type":"ContainerStarted","Data":"eb5f472b8184db7eb35b08f89a03f3dc14dee8104f7471287a1f169bb41b7635"}
Oct 30 02:59:41.622169 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:41.622103    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 02:59:41.660592 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:41.660546    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45" podStartSLOduration=1.66053205 podStartE2EDuration="1.66053205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:41.656463121 +0000 UTC m=+1126.958641485" watchObservedRunningTime="2026-10-30 02:59:41.66053205 +0000 UTC m=+1126.962710414"
Oct 30 02:59:42.642872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:42.642828    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:42.776329 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:42.776295    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:42.796897 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:42.796868    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"]
Oct 30 02:59:42.801136 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:42.801115    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 02:59:42.815887 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:42.815868    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-adca906b-dockercfg-gg64l\""
Oct 30 02:59:42.819783 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:42.819760    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"]
Oct 30 02:59:42.826449 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:42.826415    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-adca906b\""
Oct 30 02:59:42.900910 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:42.900788    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s\" (UID: \"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 02:59:42.900910 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:42.900870    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwlrt\" (UniqueName: \"kubernetes.io/projected/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-kube-api-access-pwlrt\") pod \"payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s\" (UID: \"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 02:59:43.003050 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:43.003009    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s\" (UID: \"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 02:59:43.003252 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:43.003071    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlrt\" (UniqueName: \"kubernetes.io/projected/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-kube-api-access-pwlrt\") pod \"payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s\" (UID: \"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 02:59:43.004274 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:43.004246    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s\" (UID: \"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 02:59:43.021352 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:43.021318    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwlrt\" (UniqueName: \"kubernetes.io/projected/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-kube-api-access-pwlrt\") pod \"payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s\" (UID: \"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 02:59:43.115200 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:43.115162    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 02:59:43.378599 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:43.378563    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:43.439903 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:59:43.439863    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d3756c_e5ec_41d4_b14f_ac1a9e7a3669.slice/crio-43583c6194070073962257fb49d21c5a225569f6746f7501f7d292d14fcc2ebe WatchSource:0}: Error finding container 43583c6194070073962257fb49d21c5a225569f6746f7501f7d292d14fcc2ebe: Status 404 returned error can't find the container with id 43583c6194070073962257fb49d21c5a225569f6746f7501f7d292d14fcc2ebe
Oct 30 02:59:43.444665 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:43.444636    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"]
Oct 30 02:59:43.640297 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:43.640248    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s" event={"ID":"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669","Type":"ContainerStarted","Data":"9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c"}
Oct 30 02:59:43.640488 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:43.640311    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s" event={"ID":"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669","Type":"ContainerStarted","Data":"43583c6194070073962257fb49d21c5a225569f6746f7501f7d292d14fcc2ebe"}
Oct 30 02:59:43.640488 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:43.640337    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 02:59:43.676073 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:43.676027    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s" podStartSLOduration=1.6760136939999999 podStartE2EDuration="1.676013694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:43.673602802 +0000 UTC m=+1128.975781166" watchObservedRunningTime="2026-10-30 02:59:43.676013694 +0000 UTC m=+1128.978192109"
Oct 30 02:59:44.482571 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.482540    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"]
Oct 30 02:59:44.487081 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.487057    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.530993 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.530951    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"]
Oct 30 02:59:44.553569 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.553536    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.553711 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.553606    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.553767 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.553717    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.553828 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.553813    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d85b07-775f-4fba-a17e-329a73fec898-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.553922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.553882    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czp2d\" (UniqueName: \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-kube-api-access-czp2d\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.554030 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.554009    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.554082 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.554059    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.554150 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.554130    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d85b07-775f-4fba-a17e-329a73fec898-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.554244 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.554230    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.615434 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.615385    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:44.655823 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.655793    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.655990 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.655892    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.655990 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.655944    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.656096 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.655999    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d85b07-775f-4fba-a17e-329a73fec898-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.656096 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.656026    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czp2d\" (UniqueName: \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-kube-api-access-czp2d\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.656198 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.656103    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.656198 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.656158    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.656302 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.656200    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d85b07-775f-4fba-a17e-329a73fec898-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.656302 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.656260    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.656302 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.656276    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.656302 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.656279    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.656561 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.656532    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.656668 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.656545    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.656936 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.656917    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d85b07-775f-4fba-a17e-329a73fec898-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.658219 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.658192    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.658877 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.658834    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d85b07-775f-4fba-a17e-329a73fec898-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.679692 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.679667    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.680011 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.679927    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czp2d\" (UniqueName: \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-kube-api-access-czp2d\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.814303 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.812178    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:44.814303 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:44.812933    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:45.201825 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:45.201748    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"]
Oct 30 02:59:45.206013 ip-10-0-136-238 kubenswrapper[5061]: W1030 02:59:45.205982    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d85b07_775f_4fba_a17e_329a73fec898.slice/crio-24204e64880e0d794f59c63514e9b896993047f767ee63bbb67de5ccbf824bb5 WatchSource:0}: Error finding container 24204e64880e0d794f59c63514e9b896993047f767ee63bbb67de5ccbf824bb5: Status 404 returned error can't find the container with id 24204e64880e0d794f59c63514e9b896993047f767ee63bbb67de5ccbf824bb5
Oct 30 02:59:45.208575 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:45.208546    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:59:45.208659 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:45.208605    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:59:45.208659 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:45.208634    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 02:59:45.392187 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:45.392156    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:45.655209 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:45.655175    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" event={"ID":"48d85b07-775f-4fba-a17e-329a73fec898","Type":"ContainerStarted","Data":"126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68"}
Oct 30 02:59:45.655692 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:45.655228    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" event={"ID":"48d85b07-775f-4fba-a17e-329a73fec898","Type":"ContainerStarted","Data":"24204e64880e0d794f59c63514e9b896993047f767ee63bbb67de5ccbf824bb5"}
Oct 30 02:59:45.718830 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:45.718775    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" podStartSLOduration=1.7187601479999999 podStartE2EDuration="1.718760148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:59:45.706795333 +0000 UTC m=+1131.008973698" watchObservedRunningTime="2026-10-30 02:59:45.718760148 +0000 UTC m=+1131.020938513"
Oct 30 02:59:45.813084 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:45.813054    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:46.481818 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:46.481783    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:46.814628 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:46.814595    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:59:46.815058 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:46.814649    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 02:59:46.976993 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:46.976948    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:47.812985 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:47.812951    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:59:47.813163 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:47.813004    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 02:59:48.771266 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:48.771216    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:48.816278 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:48.813406    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:59:48.816278 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:48.813468    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 02:59:49.813069 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:49.813032    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:59:49.813504 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:49.813081    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 02:59:50.477826 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:50.477790    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:50.845922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:50.845038    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:59:50.845922 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:50.845130    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 02:59:51.829512 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:51.829472    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:59:51.864082 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:51.863939    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 02:59:52.673117 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:52.673075    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 02:59:52.822779 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:52.822744    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 02:59:52.829834 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:52.829784    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 02:59:53.119874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.086952    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:53.119874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.087023    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.183872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.183013    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.183872 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.183012    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.210641 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.210598    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.221995 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.221936    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:59:53.250694 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.222637    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 02:59:53.507113 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.507013    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:53.507385 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.507344    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.770142 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.770025    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:53.854874 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.854448    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:53.905000 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:53.904951    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.016876 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.014943    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.114617 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.114573    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.194002 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.193949    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 02:59:54.212900 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.205184    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 02:59:54.306688 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.214990    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:59:54.320869 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.308098    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:59:54.323092 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.241967    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 02:59:54.329980 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.272906    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 02:59:54.357562 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.357536    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:59:54.359463 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.227903    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.364579 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.360064    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:59:54.405920 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.403502    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.405920 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.404908    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:59:54.429230 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.427961    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.479791 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.479748    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.775919 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.713075    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 02:59:54.885045 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.885000    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:54.906873 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.885998    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:59:55.070932 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.987549    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:55.070932 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.029860    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.070932 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:54.987588    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 02:59:55.070932 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.016174    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.126475 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.126428    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.346963 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.163943    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.346963 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.224354    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.346963 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.310403    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.346963 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.338881    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 02:59:55.370962 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.370923    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 02:59:55.371448 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.371417    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.430000 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.429924    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.579871 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.408992    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.674074 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.672572    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Oct 30 02:59:55.767667 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.765290    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 02:59:55.767667 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.765355    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 02:59:55.767667 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.767084    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:55.767667 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.767159    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:59:55.767667 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.767210    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:59:55.767667 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.767251    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:59:55.767667 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.767282    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:59:55.767667 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.767319    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:59:55.767667 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.767352    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 02:59:55.793191 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.793154    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:55.818252 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.818185    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 02:59:55.818809 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.818774    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 02:59:55.819077 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.819052    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 02:59:55.820072 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.820042    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 02:59:55.830758 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.830331    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 02:59:55.857134 ip-10-0-136-238 kubenswrapper[5061]: E1030 02:59:55.857093    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 02:59:55.889865 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.885352    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 02:59:55.910415 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.910362    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 02:59:55.926478 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.925393    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 02:59:55.944614 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.944409    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 02:59:55.988063 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.987684    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 02:59:55.991618 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:55.990908    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 02:59:56.000451 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:56.000346    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 02:59:56.984783 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:56.984732    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:57.099756 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:57.099724    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:58.184281 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:58.184245    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:58.329896 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:58.329834    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:58.443525 ip-10-0-136-238 kubenswrapper[5061]: I1030 02:59:58.443451    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:01.039513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:01.039477    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:01.162014 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:01.161985    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:02.729204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:02.729169    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:02.852815 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:02.852766    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:02.967364 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:02.967328    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:04.578260 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:04.578195    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:06.204217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:06.204146    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:06.315626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:06.315592    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:06.433343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:06.433302    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:07.472509 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:07.472475    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:08.198398 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:08.198368    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:09.284989 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.284952    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"]
Oct 30 03:00:09.303890 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.301737    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"]
Oct 30 03:00:09.303890 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.301925    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:09.316141 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.316116    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:00:09.363869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.329059    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:00:09.400997 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.400967    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-999a8ab0-7w4ws\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:09.401253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.401231    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgxk7\" (UniqueName: \"kubernetes.io/projected/c1107a11-8162-46a6-ba4b-4387e19705d5-kube-api-access-pgxk7\") pod \"maas-api-revoke-sub-999a8ab0-7w4ws\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:09.401559 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.401539    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-999a8ab0-7w4ws\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:09.508037 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.505823    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-999a8ab0-7w4ws\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:09.508037 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.506020    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-999a8ab0-7w4ws\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:09.508037 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.506092    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxk7\" (UniqueName: \"kubernetes.io/projected/c1107a11-8162-46a6-ba4b-4387e19705d5-kube-api-access-pgxk7\") pod \"maas-api-revoke-sub-999a8ab0-7w4ws\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:09.508037 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.507305    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-999a8ab0-7w4ws\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:09.513144 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.513122    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-999a8ab0-7w4ws\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:09.591437 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.591405    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxk7\" (UniqueName: \"kubernetes.io/projected/c1107a11-8162-46a6-ba4b-4387e19705d5-kube-api-access-pgxk7\") pod \"maas-api-revoke-sub-999a8ab0-7w4ws\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:09.686111 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:09.686078    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:10.053657 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:10.053511    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"]
Oct 30 03:00:10.056300 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:00:10.056272    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1107a11_8162_46a6_ba4b_4387e19705d5.slice/crio-86a26d8f7356f12d746efa8d46483b7cccb28e6f21643206b45e207498732a95 WatchSource:0}: Error finding container 86a26d8f7356f12d746efa8d46483b7cccb28e6f21643206b45e207498732a95: Status 404 returned error can't find the container with id 86a26d8f7356f12d746efa8d46483b7cccb28e6f21643206b45e207498732a95
Oct 30 03:00:10.172545 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:10.172511    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws" event={"ID":"c1107a11-8162-46a6-ba4b-4387e19705d5","Type":"ContainerStarted","Data":"86a26d8f7356f12d746efa8d46483b7cccb28e6f21643206b45e207498732a95"}
Oct 30 03:00:10.490280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:10.490182    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:12.127904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:12.127873    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:12.784093 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:00:12.784061    5061 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1107a11_8162_46a6_ba4b_4387e19705d5.slice/crio-f71f41bcbb9b29bc824cc6c2a0dea7eed05d77f87ad9b0ac890525acb8616fcd.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:00:13.197571 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:13.197539    5061 generic.go:356] "Generic (PLEG): container finished" podID="c1107a11-8162-46a6-ba4b-4387e19705d5" containerID="f71f41bcbb9b29bc824cc6c2a0dea7eed05d77f87ad9b0ac890525acb8616fcd" exitCode=0
Oct 30 03:00:13.197997 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:13.197628    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws" event={"ID":"c1107a11-8162-46a6-ba4b-4387e19705d5","Type":"ContainerDied","Data":"f71f41bcbb9b29bc824cc6c2a0dea7eed05d77f87ad9b0ac890525acb8616fcd"}
Oct 30 03:00:13.627473 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:13.627438    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:14.372690 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.372666    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:14.479185 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.479105    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1107a11-8162-46a6-ba4b-4387e19705d5-kube-api-access-pgxk7\" (UniqueName: \"kubernetes.io/projected/c1107a11-8162-46a6-ba4b-4387e19705d5-kube-api-access-pgxk7\") pod \"c1107a11-8162-46a6-ba4b-4387e19705d5\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") "
Oct 30 03:00:14.479350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.479217    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-openshift-ca\") pod \"c1107a11-8162-46a6-ba4b-4387e19705d5\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") "
Oct 30 03:00:14.479350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.479297    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-xks-ca\") pod \"c1107a11-8162-46a6-ba4b-4387e19705d5\" (UID: \"c1107a11-8162-46a6-ba4b-4387e19705d5\") "
Oct 30 03:00:14.479626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.479598    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-openshift-ca" pod "c1107a11-8162-46a6-ba4b-4387e19705d5" (UID: "c1107a11-8162-46a6-ba4b-4387e19705d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:14.479708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.479692    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-openshift-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:00:14.481407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.481377    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-xks-ca" pod "c1107a11-8162-46a6-ba4b-4387e19705d5" (UID: "c1107a11-8162-46a6-ba4b-4387e19705d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:00:14.481503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.481419    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1107a11-8162-46a6-ba4b-4387e19705d5-kube-api-access-pgxk7" pod "c1107a11-8162-46a6-ba4b-4387e19705d5" (UID: "c1107a11-8162-46a6-ba4b-4387e19705d5"). InnerVolumeSpecName "kube-api-access-pgxk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:14.571989 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.571959    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"]
Oct 30 03:00:14.572638 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.572622    5061 cpu_manager.go:414] "containerMap: removing container" podUID="c1107a11-8162-46a6-ba4b-4387e19705d5" containerName="revoke-keys"
Oct 30 03:00:14.572692 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.572640    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1107a11-8162-46a6-ba4b-4387e19705d5" containerName="revoke-keys"
Oct 30 03:00:14.572774 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.572764    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1107a11-8162-46a6-ba4b-4387e19705d5" containerName="revoke-keys"
Oct 30 03:00:14.576384 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.576357    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.580665 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.580644    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1107a11-8162-46a6-ba4b-4387e19705d5-maas-api-xks-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:00:14.580665 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.580666    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgxk7\" (UniqueName: \"kubernetes.io/projected/c1107a11-8162-46a6-ba4b-4387e19705d5-kube-api-access-pgxk7\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:00:14.599277 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.599245    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"]
Oct 30 03:00:14.681905 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.681841    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.682078 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.681963    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xlz\" (UniqueName: \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-kube-api-access-g7xlz\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.682078 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.682063    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.682206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.682162    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.682280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.682257    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.682320 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.682289    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.682371 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.682318    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.682411 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.682369    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.682411 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.682401    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.783812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.783737    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.783812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.783771    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.783812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.783794    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.784112 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.783813    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.784112 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.783833    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.784112 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.783923    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.784112 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.784013    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xlz\" (UniqueName: \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-kube-api-access-g7xlz\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.784112 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.784096    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.784369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.784138    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-certs\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.784369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.784198    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.784369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.784242    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-data\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.784531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.784436    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.784531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.784508    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-credential-socket\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.784879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.784836    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istiod-ca-cert\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.786114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.786094    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-envoy\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.786507 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.786491    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-podinfo\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.810660 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.810631    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xlz\" (UniqueName: \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-kube-api-access-g7xlz\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.810755 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.810717    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-token\") pod \"e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") " pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:14.890662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:14.890630    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:15.213301 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:15.213277    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"]
Oct 30 03:00:15.215098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:15.215076    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"
Oct 30 03:00:15.215199 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:15.215108    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws" event={"ID":"c1107a11-8162-46a6-ba4b-4387e19705d5","Type":"ContainerDied","Data":"86a26d8f7356f12d746efa8d46483b7cccb28e6f21643206b45e207498732a95"}
Oct 30 03:00:15.215199 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:15.215162    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86a26d8f7356f12d746efa8d46483b7cccb28e6f21643206b45e207498732a95"
Oct 30 03:00:15.216680 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:00:15.216660    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed6baf0_c209_4365_9e4c_bbd985b90a9c.slice/crio-a977f50a1e049c0b9a4aa5652662b504d0a3ff753c9bf1b9857f687b741b81ad WatchSource:0}: Error finding container a977f50a1e049c0b9a4aa5652662b504d0a3ff753c9bf1b9857f687b741b81ad: Status 404 returned error can't find the container with id a977f50a1e049c0b9a4aa5652662b504d0a3ff753c9bf1b9857f687b741b81ad
Oct 30 03:00:15.219551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:15.219526    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:00:15.219621 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:15.219591    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:00:15.219621 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:15.219616    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:00:15.483616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:15.483531    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:16.247004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:16.246968    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" event={"ID":"aed6baf0-c209-4365-9e4c-bbd985b90a9c","Type":"ContainerStarted","Data":"7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7"}
Oct 30 03:00:16.247188 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:16.247037    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" event={"ID":"aed6baf0-c209-4365-9e4c-bbd985b90a9c","Type":"ContainerStarted","Data":"a977f50a1e049c0b9a4aa5652662b504d0a3ff753c9bf1b9857f687b741b81ad"}
Oct 30 03:00:16.299841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:16.299791    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" podStartSLOduration=2.299777611 podStartE2EDuration="2.299777611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:16.295530823 +0000 UTC m=+1161.597709189" watchObservedRunningTime="2026-10-30 03:00:16.299777611 +0000 UTC m=+1161.601955976"
Oct 30 03:00:16.891868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:16.891809    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:16.893838 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:16.893811    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 03:00:16.893983 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:16.893868    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 03:00:17.083856 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:17.083807    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:17.279872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:17.279770    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:17.414005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:17.413968    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:17.891761 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:17.891723    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 03:00:17.891992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:17.891782    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 03:00:18.448627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:18.448579    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:18.584824 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:18.584789    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:18.718372 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:18.718297    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:18.891516 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:18.891479    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 03:00:18.891709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:18.891561    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 03:00:19.145474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:19.145443    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:19.892007 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:19.891976    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 03:00:19.892235 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:19.892042    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 03:00:20.901126 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:20.901092    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 03:00:20.901654 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:20.901623    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 03:00:21.332217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:21.332184    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:21.891767 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:21.891735    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 03:00:21.891998 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:21.891792    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 03:00:22.895862 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:22.895828    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:22.896241 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:22.896224    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:22.896937 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:22.896917    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:00:23.408121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:23.408088    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:25.274273 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:25.274230    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:25.460800 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:00:25.460761    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:00:27.386692 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:27.386660    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:31.074487 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:31.074449    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:32.421012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:32.420979    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:32.834572 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:32.834531    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:32.957857 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:32.957812    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:33.175330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:33.175253    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:33.675768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:33.675740    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:33.806506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:33.806475    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:34.831659 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:34.831624    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:36.516308 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:36.516270    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:36.886451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:36.886413    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:38.308715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:38.308676    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:38.719399 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:38.719315    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"]
Oct 30 03:00:38.843492 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:38.843425    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:38.956091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:38.956043    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:39.510371 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:39.510337    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:39.823678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:39.823640    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:39.951620 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:39.951579    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:40.069739 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:40.069705    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:40.380828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:40.380800    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:42.150622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:42.150583    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:43.719985 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:43.719942    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45" podUID="11a1f64d-e684-429a-899b-7f5d0b45612c" containerName="payload-pre-processing" containerID="cri-o://87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a" gracePeriod=25
Oct 30 03:00:43.811522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:43.811492    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:43.994561 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:43.994534    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 03:00:44.126110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.126068    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a1f64d-e684-429a-899b-7f5d0b45612c-kube-api-access-vzj49\" (UniqueName: \"kubernetes.io/projected/11a1f64d-e684-429a-899b-7f5d0b45612c-kube-api-access-vzj49\") pod \"11a1f64d-e684-429a-899b-7f5d0b45612c\" (UID: \"11a1f64d-e684-429a-899b-7f5d0b45612c\") "
Oct 30 03:00:44.126320 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.126240    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a1f64d-e684-429a-899b-7f5d0b45612c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a1f64d-e684-429a-899b-7f5d0b45612c-plugins-config-volume\") pod \"11a1f64d-e684-429a-899b-7f5d0b45612c\" (UID: \"11a1f64d-e684-429a-899b-7f5d0b45612c\") "
Oct 30 03:00:44.126777 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.126744    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a1f64d-e684-429a-899b-7f5d0b45612c-plugins-config-volume" pod "11a1f64d-e684-429a-899b-7f5d0b45612c" (UID: "11a1f64d-e684-429a-899b-7f5d0b45612c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:44.127059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.126950    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a1f64d-e684-429a-899b-7f5d0b45612c-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:00:44.128594 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.128562    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a1f64d-e684-429a-899b-7f5d0b45612c-kube-api-access-vzj49" pod "11a1f64d-e684-429a-899b-7f5d0b45612c" (UID: "11a1f64d-e684-429a-899b-7f5d0b45612c"). InnerVolumeSpecName "kube-api-access-vzj49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:44.228055 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.227963    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzj49\" (UniqueName: \"kubernetes.io/projected/11a1f64d-e684-429a-899b-7f5d0b45612c-kube-api-access-vzj49\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:00:44.667383 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.667339    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:44.671795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.671757    5061 generic.go:356] "Generic (PLEG): container finished" podID="11a1f64d-e684-429a-899b-7f5d0b45612c" containerID="87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a" exitCode=0
Oct 30 03:00:44.671987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.671926    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45" event={"ID":"11a1f64d-e684-429a-899b-7f5d0b45612c","Type":"ContainerDied","Data":"87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a"}
Oct 30 03:00:44.671987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.671978    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"
Oct 30 03:00:44.672121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.672010    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45" event={"ID":"11a1f64d-e684-429a-899b-7f5d0b45612c","Type":"ContainerDied","Data":"eb5f472b8184db7eb35b08f89a03f3dc14dee8104f7471287a1f169bb41b7635"}
Oct 30 03:00:44.672121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.672051    5061 scope.go:122] "RemoveContainer" containerID="87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a"
Oct 30 03:00:44.684068 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.683715    5061 scope.go:122] "RemoveContainer" containerID="87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a"
Oct 30 03:00:44.684231 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:00:44.684130    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a\": container with ID starting with 87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a not found: ID does not exist" containerID="87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a"
Oct 30 03:00:44.684231 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.684156    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a"} err="failed to get container status \"87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a\": rpc error: code = NotFound desc = could not find container \"87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a\": container with ID starting with 87c0a21a847290ea7e4bba442159b391770cd0b9c1c176f2118f0399668b1d0a not found: ID does not exist"
Oct 30 03:00:44.720274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.720245    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"]
Oct 30 03:00:44.730708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.730678    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93fe9de7-6b46855855-l6d45"]
Oct 30 03:00:44.849201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.849170    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a1f64d-e684-429a-899b-7f5d0b45612c" path="/var/lib/kubelet/pods/11a1f64d-e684-429a-899b-7f5d0b45612c/volumes"
Oct 30 03:00:44.919126 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:44.919045    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:45.152836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:45.152797    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:45.625972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:45.625938    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:45.840338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:45.840308    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:45.975331 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:45.975240    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:46.166418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:46.166385    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:46.607644 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:46.607611    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:46.824902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:46.824834    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:46.938470 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:46.938391    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:47.126698 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:47.126665    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:47.276906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:47.276802    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:48.146885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:48.138565    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:48.402925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:48.402178    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"]
Oct 30 03:00:48.526888 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:48.526838    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:48.656936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:48.649358    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:49.081886 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:49.073250    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:49.262874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:49.238468    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:49.498044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:49.497970    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:50.467087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:50.467056    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:50.730229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:50.730143    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:50.860952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:50.860915    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:51.775372 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:51.775339    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:52.914421 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:52.914387    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:53.026255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.026220    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:53.350168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.350135    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:53.403340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.403307    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s" podUID="88d3756c-e5ec-41d4-b14f-ac1a9e7a3669" containerName="payload-pre-processing" containerID="cri-o://9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c" gracePeriod=25
Oct 30 03:00:53.713259 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.713234    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 03:00:53.732177 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.732121    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:53.821677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.821641    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-kube-api-access-pwlrt\" (UniqueName: \"kubernetes.io/projected/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-kube-api-access-pwlrt\") pod \"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669\" (UID: \"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669\") "
Oct 30 03:00:53.821891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.821777    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-plugins-config-volume\") pod \"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669\" (UID: \"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669\") "
Oct 30 03:00:53.822724 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.822673    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-plugins-config-volume" pod "88d3756c-e5ec-41d4-b14f-ac1a9e7a3669" (UID: "88d3756c-e5ec-41d4-b14f-ac1a9e7a3669"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:00:53.824300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.824275    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-kube-api-access-pwlrt" pod "88d3756c-e5ec-41d4-b14f-ac1a9e7a3669" (UID: "88d3756c-e5ec-41d4-b14f-ac1a9e7a3669"). InnerVolumeSpecName "kube-api-access-pwlrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:00:53.883838 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.883807    5061 generic.go:356] "Generic (PLEG): container finished" podID="88d3756c-e5ec-41d4-b14f-ac1a9e7a3669" containerID="9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c" exitCode=0
Oct 30 03:00:53.884027 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.883889    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"
Oct 30 03:00:53.884027 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.883906    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s" event={"ID":"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669","Type":"ContainerDied","Data":"9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c"}
Oct 30 03:00:53.884027 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.883967    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s" event={"ID":"88d3756c-e5ec-41d4-b14f-ac1a9e7a3669","Type":"ContainerDied","Data":"43583c6194070073962257fb49d21c5a225569f6746f7501f7d292d14fcc2ebe"}
Oct 30 03:00:53.884027 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.883998    5061 scope.go:122] "RemoveContainer" containerID="9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c"
Oct 30 03:00:53.894154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.894122    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:53.897224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.897205    5061 scope.go:122] "RemoveContainer" containerID="9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c"
Oct 30 03:00:53.897498 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:00:53.897476    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c\": container with ID starting with 9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c not found: ID does not exist" containerID="9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c"
Oct 30 03:00:53.897551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.897508    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c"} err="failed to get container status \"9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c\": rpc error: code = NotFound desc = could not find container \"9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c\": container with ID starting with 9beeab5f75591d9757bb73c8edd7db5dac928183030f853730871c8b25e22d8c not found: ID does not exist"
Oct 30 03:00:53.924078 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.924051    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwlrt\" (UniqueName: \"kubernetes.io/projected/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-kube-api-access-pwlrt\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:00:53.924078 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.924077    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:00:53.936240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.936160    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"]
Oct 30 03:00:53.944908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.944881    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-adca906b-f99ff57d7-nnq2s"]
Oct 30 03:00:53.982637 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.982600    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"]
Oct 30 03:00:53.983656 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.983634    5061 cpu_manager.go:414] "containerMap: removing container" podUID="88d3756c-e5ec-41d4-b14f-ac1a9e7a3669" containerName="payload-pre-processing"
Oct 30 03:00:53.983718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.983661    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d3756c-e5ec-41d4-b14f-ac1a9e7a3669" containerName="payload-pre-processing"
Oct 30 03:00:53.983718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.983677    5061 cpu_manager.go:414] "containerMap: removing container" podUID="11a1f64d-e684-429a-899b-7f5d0b45612c" containerName="payload-pre-processing"
Oct 30 03:00:53.983718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.983685    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a1f64d-e684-429a-899b-7f5d0b45612c" containerName="payload-pre-processing"
Oct 30 03:00:53.983923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.983908    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d3756c-e5ec-41d4-b14f-ac1a9e7a3669" containerName="payload-pre-processing"
Oct 30 03:00:53.983968 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.983926    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a1f64d-e684-429a-899b-7f5d0b45612c" containerName="payload-pre-processing"
Oct 30 03:00:53.989395 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:53.989373    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:00:54.037973 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.037926    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"]
Oct 30 03:00:54.061233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.061199    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9ca6fbc2\""
Oct 30 03:00:54.061406 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.061392    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9ca6fbc2-dockercfg-wvg9l\""
Oct 30 03:00:54.062739 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.062708    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:54.128323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.128288    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj8zz\" (UniqueName: \"kubernetes.io/projected/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-kube-api-access-pj8zz\") pod \"payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj\" (UID: \"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a\") " pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:00:54.128508 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.128417    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj\" (UID: \"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a\") " pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:00:54.230423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.230320    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj8zz\" (UniqueName: \"kubernetes.io/projected/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-kube-api-access-pj8zz\") pod \"payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj\" (UID: \"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a\") " pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:00:54.231421 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.230450    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj\" (UID: \"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a\") " pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:00:54.231571 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.231546    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj\" (UID: \"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a\") " pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:00:54.250160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.250133    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj8zz\" (UniqueName: \"kubernetes.io/projected/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-kube-api-access-pj8zz\") pod \"payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj\" (UID: \"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a\") " pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:00:54.274894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.274836    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:54.365052 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.365015    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:00:54.691779 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.691532    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"]
Oct 30 03:00:54.695158 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:00:54.695121    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7689db8c_81c2_4e2e_81f6_e0b10fd4ae0a.slice/crio-e7731935e79ff07a681c7a099d01b90be757a635af2e2c3a90345992fc7088de WatchSource:0}: Error finding container e7731935e79ff07a681c7a099d01b90be757a635af2e2c3a90345992fc7088de: Status 404 returned error can't find the container with id e7731935e79ff07a681c7a099d01b90be757a635af2e2c3a90345992fc7088de
Oct 30 03:00:54.849754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.849725    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d3756c-e5ec-41d4-b14f-ac1a9e7a3669" path="/var/lib/kubelet/pods/88d3756c-e5ec-41d4-b14f-ac1a9e7a3669/volumes"
Oct 30 03:00:54.893233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.892962    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj" event={"ID":"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a","Type":"ContainerStarted","Data":"05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593"}
Oct 30 03:00:54.893233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.893032    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj" event={"ID":"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a","Type":"ContainerStarted","Data":"e7731935e79ff07a681c7a099d01b90be757a635af2e2c3a90345992fc7088de"}
Oct 30 03:00:54.893233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.893067    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:00:54.935470 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.935411    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj" podStartSLOduration=1.935390454 podStartE2EDuration="1.935390454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:54.928421491 +0000 UTC m=+1200.230599857" watchObservedRunningTime="2026-10-30 03:00:54.935390454 +0000 UTC m=+1200.237568822"
Oct 30 03:00:54.994352 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:54.994252    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:55.006871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:55.006033    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:00:55.006871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:55.006633    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:00:55.009352 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:55.009330    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:00:55.010034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:55.010017    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:00:55.193201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:55.193170    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:55.329705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:55.329655    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:55.452570 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:00:55.452536    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:00:56.807980 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:56.807935    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:57.661171 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:57.661134    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:57.797306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:57.797261    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:57.922221 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:57.922109    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:58.057724 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:58.057682    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:58.637315 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:58.637277    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:59.330144 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:59.330101    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:59.532877 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:00:59.528312    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:01.009044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:01.009003    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:01.127042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:01.127011    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:01.424928 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:01.424894    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:03.088109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:03.088079    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:04.817221 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:04.817179    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:05.904994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:05.904960    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:01:06.970839 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:06.970791    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:08.379449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:08.379407    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:08.599100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:08.599063    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:10.190742 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:10.190711    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:10.306971 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:10.306935    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:10.422471 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:10.422432    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:11.836054 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:11.836019    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:13.583892 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:13.583839    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:14.371032 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:14.371000    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:15.341832 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:15.341797    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:15.450912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:15.450879    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:15.565795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:15.565759    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:17.813986 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:17.813950    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:20.225368 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:20.225329    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:22.021588 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:22.021551    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:22.541588 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:22.541550    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:23.974271 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:23.974235    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:24.114078 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:24.114039    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:24.227310 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:24.227204    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:24.985213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:24.985181    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:25.446311 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:01:25.446275    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:01:25.885937 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:25.885884    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:28.219689 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.219649    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:28.801231 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.801193    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"]
Oct 30 03:01:28.806167 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.806145    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:28.836909 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.836882    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"]
Oct 30 03:01:28.855261 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.855233    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"]
Oct 30 03:01:28.860426 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.860401    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:28.865442 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.865421    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"]
Oct 30 03:01:28.908196 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.908166    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:28.908355 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.908213    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:28.908426 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.908364    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:28.908480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.908429    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91dfc434-1255-4315-a28d-b85d3f649964-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:28.908567 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.908540    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:28.908687 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.908661    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:28.908787 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.908767    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91dfc434-1255-4315-a28d-b85d3f649964-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:28.908913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.908836    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sszlp\" (UniqueName: \"kubernetes.io/projected/91dfc434-1255-4315-a28d-b85d3f649964-kube-api-access-sszlp\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:28.909046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.908978    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:28.909046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.909016    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:28.909170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.909076    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:28.909170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.909143    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:28.909269 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.909168    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91dfc434-1255-4315-a28d-b85d3f649964-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:28.909319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.909285    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcjrb\" (UniqueName: \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-kube-api-access-bcjrb\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:28.909375 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.909346    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:28.909427 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.909395    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:28.909477 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.909427    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:28.909531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:28.909479    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.011738 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.011710    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.011921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.011752    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.011921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.011774    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.011921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.011792    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.011921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.011809    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.011921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.011866    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.012206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.011978    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91dfc434-1255-4315-a28d-b85d3f649964-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.012206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012038    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.012206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012107    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.012206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012171    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91dfc434-1255-4315-a28d-b85d3f649964-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.012206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012179    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.012453 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012239    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sszlp\" (UniqueName: \"kubernetes.io/projected/91dfc434-1255-4315-a28d-b85d3f649964-kube-api-access-sszlp\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.012453 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012258    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.012453 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012293    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.012453 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012319    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.012453 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012343    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.012453 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012405    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.012453 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012430    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91dfc434-1255-4315-a28d-b85d3f649964-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.012833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012455    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.012833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012517    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcjrb\" (UniqueName: \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-kube-api-access-bcjrb\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.012833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012587    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.012833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012707    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91dfc434-1255-4315-a28d-b85d3f649964-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.013134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012886    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.013134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012970    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.013134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.012988    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.013295 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.013146    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.013295 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.013230    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.013403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.013338    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.014343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.014321    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91dfc434-1255-4315-a28d-b85d3f649964-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.014860 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.014825    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.014946 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.014903    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91dfc434-1255-4315-a28d-b85d3f649964-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.015051 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.015035    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.030023 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.029995    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.030679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.030652    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91dfc434-1255-4315-a28d-b85d3f649964-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.030679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.030668    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcjrb\" (UniqueName: \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-kube-api-access-bcjrb\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ngtcs\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.031174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.031150    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sszlp\" (UniqueName: \"kubernetes.io/projected/91dfc434-1255-4315-a28d-b85d3f649964-kube-api-access-sszlp\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-54ldl\" (UID: \"91dfc434-1255-4315-a28d-b85d3f649964\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.120232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.120141    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:29.178580 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.178551    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:29.437122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.437100    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"]
Oct 30 03:01:29.439738 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:01:29.439713    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d985ff0_fd99_4f30_8e24_e9122ac36f88.slice/crio-10e1ba17a12fde256db9808138582f2e643e4eb630dd5e1e31a11f58fb996429 WatchSource:0}: Error finding container 10e1ba17a12fde256db9808138582f2e643e4eb630dd5e1e31a11f58fb996429: Status 404 returned error can't find the container with id 10e1ba17a12fde256db9808138582f2e643e4eb630dd5e1e31a11f58fb996429
Oct 30 03:01:29.442244 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.442212    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:29.442343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.442272    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:29.442343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.442298    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:29.513033 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.513007    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"]
Oct 30 03:01:29.513148 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:01:29.513110    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91dfc434_1255_4315_a28d_b85d3f649964.slice/crio-3b03f446ab05a6f1d7ea5d561a0ffe72e806b705754c86f46f265901b83b3399 WatchSource:0}: Error finding container 3b03f446ab05a6f1d7ea5d561a0ffe72e806b705754c86f46f265901b83b3399: Status 404 returned error can't find the container with id 3b03f446ab05a6f1d7ea5d561a0ffe72e806b705754c86f46f265901b83b3399
Oct 30 03:01:29.516118 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.516082    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:29.516207 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.516163    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:29.516207 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:29.516201    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:30.309944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:30.309909    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" event={"ID":"91dfc434-1255-4315-a28d-b85d3f649964","Type":"ContainerStarted","Data":"5f114eba595903db34ea7dc237bdd53b84b001d5a12e4ce7ff66ecdb5f92b661"}
Oct 30 03:01:30.310153 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:30.309965    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" event={"ID":"91dfc434-1255-4315-a28d-b85d3f649964","Type":"ContainerStarted","Data":"3b03f446ab05a6f1d7ea5d561a0ffe72e806b705754c86f46f265901b83b3399"}
Oct 30 03:01:30.311705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:30.311678    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" event={"ID":"6d985ff0-fd99-4f30-8e24-e9122ac36f88","Type":"ContainerStarted","Data":"0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7"}
Oct 30 03:01:30.311819 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:30.311729    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" event={"ID":"6d985ff0-fd99-4f30-8e24-e9122ac36f88","Type":"ContainerStarted","Data":"10e1ba17a12fde256db9808138582f2e643e4eb630dd5e1e31a11f58fb996429"}
Oct 30 03:01:30.384871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:30.383789    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" podStartSLOduration=2.383758236 podStartE2EDuration="2.383758236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:30.370341875 +0000 UTC m=+1235.672520240" watchObservedRunningTime="2026-10-30 03:01:30.383758236 +0000 UTC m=+1235.685936603"
Oct 30 03:01:30.412759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:30.412516    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" podStartSLOduration=2.412503736 podStartE2EDuration="2.412503736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:30.411140603 +0000 UTC m=+1235.713318967" watchObservedRunningTime="2026-10-30 03:01:30.412503736 +0000 UTC m=+1235.714682100"
Oct 30 03:01:30.759542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:30.759408    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:30.976509 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:30.976473    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:31.136872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:31.136411    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:31.136872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:31.136583    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 03:01:31.136872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:31.136638    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 03:01:31.183872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:31.183023    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:31.183872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:31.183173    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-54ldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:01:31.183872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:31.183230    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" podUID="91dfc434-1255-4315-a28d-b85d3f649964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 03:01:31.384767 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:31.384573    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:31.517381 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:31.517302    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:32.126872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:32.126162    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 03:01:32.126872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:32.126220    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 03:01:32.179176 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:32.179145    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-54ldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:01:32.179338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:32.179194    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" podUID="91dfc434-1255-4315-a28d-b85d3f649964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 03:01:33.140870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:33.123199    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 03:01:33.140870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:33.123259    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 03:01:33.140870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:33.128337    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:33.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:33.190729    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-54ldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:01:33.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:33.190811    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" podUID="91dfc434-1255-4315-a28d-b85d3f649964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 03:01:34.125873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:34.123208    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 03:01:34.125873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:34.123270    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 03:01:34.178901 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:34.178869    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-54ldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:01:34.179297 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:34.178923    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" podUID="91dfc434-1255-4315-a28d-b85d3f649964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 03:01:34.790142 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:34.790111    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:35.139873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:35.133202    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 03:01:35.139873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:35.133263    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 03:01:35.179537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:35.179490    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-54ldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:01:35.179968 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:35.179552    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" podUID="91dfc434-1255-4315-a28d-b85d3f649964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 03:01:36.002326 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:36.002279    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:36.120579 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:36.120542    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 03:01:36.120786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:36.120604    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 03:01:36.179024 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:36.178991    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-54ldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:01:36.179215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:36.179060    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" podUID="91dfc434-1255-4315-a28d-b85d3f649964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 03:01:37.120501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:37.120468    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 03:01:37.121034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:37.120527    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 03:01:37.179620 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:37.179585    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-54ldl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:01:37.179779 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:37.179645    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" podUID="91dfc434-1255-4315-a28d-b85d3f649964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 03:01:38.220631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:38.220594    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:38.511653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:38.511562    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:38.512246 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:38.512202    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:38.513147 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:38.513127    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl"
Oct 30 03:01:38.548339 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:38.548310    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:38.548639 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:38.548615    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:38.549507 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:38.549487    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:01:41.137855 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:41.137816    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:42.087567 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:42.087532    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:43.143071 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:43.142995    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:43.270284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:43.270247    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:44.908370 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:44.908337    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:45.093743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:45.093705    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:45.616742 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:45.616707    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:46.071391 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.071361    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"]
Oct 30 03:01:46.076677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.076654    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.091977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.091954    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-360818-openshift-default-dockercfg-clvks\""
Oct 30 03:01:46.106382 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.106359    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"]
Oct 30 03:01:46.160067 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.160034    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:46.163534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.163503    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.163698 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.163681    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.163768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.163748    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.163862 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.163796    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.163940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.163916    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.163988 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.163960    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksrk4\" (UniqueName: \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-kube-api-access-ksrk4\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.164151 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.164127    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.164193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.164174    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.164312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.164290    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3f605d-7d6f-4e51-ada8-12357911adc7-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.280871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.265816    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3f605d-7d6f-4e51-ada8-12357911adc7-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.307909 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.296067    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.311242 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.311201    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.325072 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.324994    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.325618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.325592    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.326214 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.326193    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.326728 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.326710    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksrk4\" (UniqueName: \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-kube-api-access-ksrk4\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.343869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.327318    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.345026 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.344997    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.345340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.326122    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.345482 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.283989    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3f605d-7d6f-4e51-ada8-12357911adc7-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.345584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.326635    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.345678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.322272    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.345759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.310454    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.345865 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.331982    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.345997 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.310881    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:46.359870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.355371    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.379692 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.379659    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksrk4\" (UniqueName: \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-kube-api-access-ksrk4\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.380387 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.380361    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.403869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.397282    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:46.724746 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.724663    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:46.750437 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.750411    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"]
Oct 30 03:01:46.753931 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:01:46.753903    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3f605d_7d6f_4e51_ada8_12357911adc7.slice/crio-c643caddb9b0e76643f3827e27b83017e8e85947bdcbe1c82e554f5ffc16c42e WatchSource:0}: Error finding container c643caddb9b0e76643f3827e27b83017e8e85947bdcbe1c82e554f5ffc16c42e: Status 404 returned error can't find the container with id c643caddb9b0e76643f3827e27b83017e8e85947bdcbe1c82e554f5ffc16c42e
Oct 30 03:01:46.756377 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.756345    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:46.756474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.756411    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:46.756474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:46.756452    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:47.084746 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:47.084706    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:47.496599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:47.496520    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" event={"ID":"4a3f605d-7d6f-4e51-ada8-12357911adc7","Type":"ContainerStarted","Data":"c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265"}
Oct 30 03:01:47.496599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:47.496577    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" event={"ID":"4a3f605d-7d6f-4e51-ada8-12357911adc7","Type":"ContainerStarted","Data":"c643caddb9b0e76643f3827e27b83017e8e85947bdcbe1c82e554f5ffc16c42e"}
Oct 30 03:01:47.556757 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:47.556708    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" podStartSLOduration=1.556691896 podStartE2EDuration="1.556691896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:47.546302553 +0000 UTC m=+1252.848480918" watchObservedRunningTime="2026-10-30 03:01:47.556691896 +0000 UTC m=+1252.858870262"
Oct 30 03:01:48.398065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:48.398035    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:48.403388 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:48.403360    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:48.503631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:48.503595    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:48.504578 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:48.504555    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:01:48.682167 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:48.682085    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"]
Oct 30 03:01:48.863606 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:48.863567    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:49.011126 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:49.010808    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:49.170546 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:49.170510    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:50.112152 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:50.112123    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:50.250926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:50.250885    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:50.378908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:50.378791    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:51.499573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:51.499540    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:52.058973 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.058935    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:52.185981 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.185950    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:52.210135 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.210106    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"]
Oct 30 03:01:52.215055 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.215038    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:01:52.230471 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.230447    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"]
Oct 30 03:01:52.236996 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.236971    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-360818\""
Oct 30 03:01:52.237117 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.237006    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-360818-dockercfg-tb56m\""
Oct 30 03:01:52.243003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.242976    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc979a77-665d-40c3-85f7-66313483e56a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg\" (UID: \"bc979a77-665d-40c3-85f7-66313483e56a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:01:52.243167 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.243138    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7cr\" (UniqueName: \"kubernetes.io/projected/bc979a77-665d-40c3-85f7-66313483e56a-kube-api-access-qr7cr\") pod \"payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg\" (UID: \"bc979a77-665d-40c3-85f7-66313483e56a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:01:52.320936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.310229    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"]
Oct 30 03:01:52.320936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.317359    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:01:52.402868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.394731    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"]
Oct 30 03:01:52.410880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.407545    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc979a77-665d-40c3-85f7-66313483e56a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg\" (UID: \"bc979a77-665d-40c3-85f7-66313483e56a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:01:52.410880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.407604    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2\" (UID: \"0dbca485-e20b-4e49-9aa4-3cac4eb994c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:01:52.410880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.407684    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7cr\" (UniqueName: \"kubernetes.io/projected/bc979a77-665d-40c3-85f7-66313483e56a-kube-api-access-qr7cr\") pod \"payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg\" (UID: \"bc979a77-665d-40c3-85f7-66313483e56a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:01:52.410880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.407751    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcn6l\" (UniqueName: \"kubernetes.io/projected/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-kube-api-access-tcn6l\") pod \"payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2\" (UID: \"0dbca485-e20b-4e49-9aa4-3cac4eb994c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:01:52.410880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.408648    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc979a77-665d-40c3-85f7-66313483e56a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg\" (UID: \"bc979a77-665d-40c3-85f7-66313483e56a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:01:52.433509 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.433476    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7cr\" (UniqueName: \"kubernetes.io/projected/bc979a77-665d-40c3-85f7-66313483e56a-kube-api-access-qr7cr\") pod \"payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg\" (UID: \"bc979a77-665d-40c3-85f7-66313483e56a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:01:52.509356 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.509322    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2\" (UID: \"0dbca485-e20b-4e49-9aa4-3cac4eb994c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:01:52.509784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.509455    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcn6l\" (UniqueName: \"kubernetes.io/projected/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-kube-api-access-tcn6l\") pod \"payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2\" (UID: \"0dbca485-e20b-4e49-9aa4-3cac4eb994c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:01:52.510296 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.510276    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2\" (UID: \"0dbca485-e20b-4e49-9aa4-3cac4eb994c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:01:52.527251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.527227    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcn6l\" (UniqueName: \"kubernetes.io/projected/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-kube-api-access-tcn6l\") pod \"payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2\" (UID: \"0dbca485-e20b-4e49-9aa4-3cac4eb994c9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:01:52.531976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.531952    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:01:52.706192 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.706158    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:01:52.857211 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:52.857185    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"]
Oct 30 03:01:53.036527 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.036506    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"]
Oct 30 03:01:53.038464 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:01:53.038444    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbca485_e20b_4e49_9aa4_3cac4eb994c9.slice/crio-fd2858c1b365c6040e65ffc8c9ee4820aae9cfcdac4307aeaeca21baa755c0c8 WatchSource:0}: Error finding container fd2858c1b365c6040e65ffc8c9ee4820aae9cfcdac4307aeaeca21baa755c0c8: Status 404 returned error can't find the container with id fd2858c1b365c6040e65ffc8c9ee4820aae9cfcdac4307aeaeca21baa755c0c8
Oct 30 03:01:53.154511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.154488    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:53.570158 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.570126    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2" event={"ID":"0dbca485-e20b-4e49-9aa4-3cac4eb994c9","Type":"ContainerStarted","Data":"74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41"}
Oct 30 03:01:53.570595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.570180    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2" event={"ID":"0dbca485-e20b-4e49-9aa4-3cac4eb994c9","Type":"ContainerStarted","Data":"fd2858c1b365c6040e65ffc8c9ee4820aae9cfcdac4307aeaeca21baa755c0c8"}
Oct 30 03:01:53.570595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.570208    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:01:53.571763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.571737    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg" event={"ID":"bc979a77-665d-40c3-85f7-66313483e56a","Type":"ContainerStarted","Data":"0b2dfb54d9662accc60a23910772b4f62aa56fce1a82a09996ad4414777216dc"}
Oct 30 03:01:53.571895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.571779    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg" event={"ID":"bc979a77-665d-40c3-85f7-66313483e56a","Type":"ContainerStarted","Data":"551bcde1b0de753e7b539227f9d236bc72c7efc811bab980192e5102624661e5"}
Oct 30 03:01:53.571895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.571864    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:01:53.605115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.605065    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2" podStartSLOduration=1.605048661 podStartE2EDuration="1.605048661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:53.603977341 +0000 UTC m=+1258.906155706" watchObservedRunningTime="2026-10-30 03:01:53.605048661 +0000 UTC m=+1258.907227027"
Oct 30 03:01:53.640268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.640213    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg" podStartSLOduration=1.6401958890000001 podStartE2EDuration="1.640195889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:53.637439309 +0000 UTC m=+1258.939617673" watchObservedRunningTime="2026-10-30 03:01:53.640195889 +0000 UTC m=+1258.942374256"
Oct 30 03:01:53.682834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.682806    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj" podUID="7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a" containerName="payload-processing" containerID="cri-o://05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593" gracePeriod=25
Oct 30 03:01:53.778257 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.778224    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:53.914352 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:53.913145    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:54.013369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.013336    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:01:54.063327 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.063297    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-plugins-config-volume\") pod \"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a\" (UID: \"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a\") "
Oct 30 03:01:54.063524 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.063455    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-kube-api-access-pj8zz\" (UniqueName: \"kubernetes.io/projected/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-kube-api-access-pj8zz\") pod \"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a\" (UID: \"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a\") "
Oct 30 03:01:54.063685 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.063658    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-plugins-config-volume" pod "7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a" (UID: "7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:01:54.064121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.064101    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:01:54.065896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.065868    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-kube-api-access-pj8zz" pod "7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a" (UID: "7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a"). InnerVolumeSpecName "kube-api-access-pj8zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:01:54.158047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.157950    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:54.166866 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.166819    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj8zz\" (UniqueName: \"kubernetes.io/projected/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a-kube-api-access-pj8zz\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:01:54.280841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.280792    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:54.582540 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.582505    5061 generic.go:356] "Generic (PLEG): container finished" podID="7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a" containerID="05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593" exitCode=0
Oct 30 03:01:54.583080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.582565    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"
Oct 30 03:01:54.583080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.582583    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj" event={"ID":"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a","Type":"ContainerDied","Data":"05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593"}
Oct 30 03:01:54.583080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.582682    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj" event={"ID":"7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a","Type":"ContainerDied","Data":"e7731935e79ff07a681c7a099d01b90be757a635af2e2c3a90345992fc7088de"}
Oct 30 03:01:54.583080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.582720    5061 scope.go:122] "RemoveContainer" containerID="05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593"
Oct 30 03:01:54.596797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.596778    5061 scope.go:122] "RemoveContainer" containerID="05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593"
Oct 30 03:01:54.597199 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:01:54.597160    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593\": container with ID starting with 05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593 not found: ID does not exist" containerID="05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593"
Oct 30 03:01:54.597318 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.597205    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593"} err="failed to get container status \"05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593\": rpc error: code = NotFound desc = could not find container \"05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593\": container with ID starting with 05eed2be9f34b391376f3f53feba1afaaf1f05730941cb0fe7afdadc2caba593 not found: ID does not exist"
Oct 30 03:01:54.630567 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.630535    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"]
Oct 30 03:01:54.639561 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.639533    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9ca6fbc2-9569bb956-ww7dj"]
Oct 30 03:01:54.849047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:54.848978    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a" path="/var/lib/kubelet/pods/7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a/volumes"
Oct 30 03:01:55.015558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:55.015528    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:55.452261 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:01:55.452220    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:01:55.493354 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:55.493325    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:55.779610 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:55.772159    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:56.313545 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:56.313517    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:58.110899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:58.110856    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:58.640016 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:58.639987    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:59.096440 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.096402    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"]
Oct 30 03:01:59.097674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.097646    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a" containerName="payload-processing"
Oct 30 03:01:59.097674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.097670    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a" containerName="payload-processing"
Oct 30 03:01:59.097931 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.097917    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7689db8c-81c2-4e2e-81f6-e0b10fd4ae0a" containerName="payload-processing"
Oct 30 03:01:59.103321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.103299    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.120114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.120082    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-ac7936-openshift-default-dockercfg-4nxgx\""
Oct 30 03:01:59.133334 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.133310    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"]
Oct 30 03:01:59.176603 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.176574    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:59.268870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.260934    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-data\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.268870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.260986    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-token\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.268870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.261027    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-envoy\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.268870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.261117    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-credential-socket\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.268870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.261148    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istiod-ca-cert\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.268870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.261205    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-podinfo\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.268870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.261249    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m75sr\" (UniqueName: \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-kube-api-access-m75sr\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.268870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.261354    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-certs\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.268870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.261434    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-socket\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.363940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.362973    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-certs\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.363940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.363056    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-socket\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.363940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.363139    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-data\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.363940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.363157    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-token\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.363940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.363181    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-envoy\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.363940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.363241    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-credential-socket\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.363940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.363259    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istiod-ca-cert\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.363940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.363298    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-podinfo\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.363940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.363327    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75sr\" (UniqueName: \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-kube-api-access-m75sr\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.363940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.363912    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-certs\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.364507 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.364117    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-socket\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.364507 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.364270    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-data\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.365213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.365185    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-credential-socket\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.365451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.365433    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istiod-ca-cert\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.366972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.366952    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-envoy\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.367342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.367321    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-podinfo\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.381967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.381940    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-token\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.382550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.382527    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75sr\" (UniqueName: \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-kube-api-access-m75sr\") pod \"e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") " pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.422501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.422465    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:01:59.504997 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.502252    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:59.761506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.761407    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:59.823727 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.823696    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"]
Oct 30 03:01:59.831006 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:01:59.830969    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab1ff26_0edb_41ef_a952_ae6a17553c4c.slice/crio-392e83b3f8d73c383a80fd2f8d0df67676bae82219949fb68102b463acf57b47 WatchSource:0}: Error finding container 392e83b3f8d73c383a80fd2f8d0df67676bae82219949fb68102b463acf57b47: Status 404 returned error can't find the container with id 392e83b3f8d73c383a80fd2f8d0df67676bae82219949fb68102b463acf57b47
Oct 30 03:01:59.833477 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.833443    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:59.833589 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.833532    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:59.833589 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.833563    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:01:59.900727 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:01:59.900678    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:00.128218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:00.128178    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:00.650688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:00.650652    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" event={"ID":"4ab1ff26-0edb-41ef-a952-ae6a17553c4c","Type":"ContainerStarted","Data":"a1b9ee68113f7a66bdffb7771cdf89e25c483d879d6da77fcbf3c909ffb3fa62"}
Oct 30 03:02:00.650901 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:00.650711    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" event={"ID":"4ab1ff26-0edb-41ef-a952-ae6a17553c4c","Type":"ContainerStarted","Data":"392e83b3f8d73c383a80fd2f8d0df67676bae82219949fb68102b463acf57b47"}
Oct 30 03:02:00.712352 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:00.712304    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" podStartSLOduration=1.712291411 podStartE2EDuration="1.712291411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:00.700629131 +0000 UTC m=+1266.002807495" watchObservedRunningTime="2026-10-30 03:02:00.712291411 +0000 UTC m=+1266.014469775"
Oct 30 03:02:01.423409 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:01.423374    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:02:01.429623 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:01.429597    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:02:01.623766 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:01.623731    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:01.659208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:01.659177    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:02:01.660354 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:01.660330    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:02:03.051768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:03.051716    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:03.178309 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:03.178281    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:03.314377 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:03.314295    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:03.528447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:03.528413    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:04.587280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:04.587242    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:02:04.587669 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:04.587635    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:02:04.882069 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:04.881836    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:05.402969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.402932    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:05.501023 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.500982    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"]
Oct 30 03:02:05.507203 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.507175    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:02:05.525063 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.525037    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"]
Oct 30 03:02:05.529438 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.529370    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-ac7936\""
Oct 30 03:02:05.536199 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.536165    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:05.575034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.574974    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q99v\" (UniqueName: \"kubernetes.io/projected/df3304e2-e0ca-482e-8f79-52e0724e4663-kube-api-access-8q99v\") pod \"payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg\" (UID: \"df3304e2-e0ca-482e-8f79-52e0724e4663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:02:05.576343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.575417    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3304e2-e0ca-482e-8f79-52e0724e4663-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg\" (UID: \"df3304e2-e0ca-482e-8f79-52e0724e4663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:02:05.677166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.677081    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q99v\" (UniqueName: \"kubernetes.io/projected/df3304e2-e0ca-482e-8f79-52e0724e4663-kube-api-access-8q99v\") pod \"payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg\" (UID: \"df3304e2-e0ca-482e-8f79-52e0724e4663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:02:05.678227 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.678198    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3304e2-e0ca-482e-8f79-52e0724e4663-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg\" (UID: \"df3304e2-e0ca-482e-8f79-52e0724e4663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:02:05.680707 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.680678    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3304e2-e0ca-482e-8f79-52e0724e4663-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg\" (UID: \"df3304e2-e0ca-482e-8f79-52e0724e4663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:02:05.699931 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.699906    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q99v\" (UniqueName: \"kubernetes.io/projected/df3304e2-e0ca-482e-8f79-52e0724e4663-kube-api-access-8q99v\") pod \"payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg\" (UID: \"df3304e2-e0ca-482e-8f79-52e0724e4663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:02:05.841210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:05.841176    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:02:06.322193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:06.316676    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"]
Oct 30 03:02:06.333423 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:02:06.333392    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3304e2_e0ca_482e_8f79_52e0724e4663.slice/crio-544d007ffb8df67851f6fc93a5589a6daabc0976be60b9264d8c9cf7c777402f WatchSource:0}: Error finding container 544d007ffb8df67851f6fc93a5589a6daabc0976be60b9264d8c9cf7c777402f: Status 404 returned error can't find the container with id 544d007ffb8df67851f6fc93a5589a6daabc0976be60b9264d8c9cf7c777402f
Oct 30 03:02:06.738681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:06.738578    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg" event={"ID":"df3304e2-e0ca-482e-8f79-52e0724e4663","Type":"ContainerStarted","Data":"f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635"}
Oct 30 03:02:06.738681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:06.738657    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:02:06.739133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:06.738683    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg" event={"ID":"df3304e2-e0ca-482e-8f79-52e0724e4663","Type":"ContainerStarted","Data":"544d007ffb8df67851f6fc93a5589a6daabc0976be60b9264d8c9cf7c777402f"}
Oct 30 03:02:06.778631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:06.778566    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg" podStartSLOduration=1.778547631 podStartE2EDuration="1.778547631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:06.773960362 +0000 UTC m=+1272.076138726" watchObservedRunningTime="2026-10-30 03:02:06.778547631 +0000 UTC m=+1272.080725995"
Oct 30 03:02:07.437872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:07.424508    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:09.537619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:09.537583    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:09.661898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:09.658963    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:09.775565 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:09.775523    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:11.422044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:11.422015    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:12.148057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:12.148025    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:13.142008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:13.141973    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:14.788650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:14.788618    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:16.370523 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:16.370491    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:17.751487 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:17.751460    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:02:18.072205 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:18.072165    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:18.189186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:18.189153    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:19.641218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:19.641185    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:19.759289 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:19.759251    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:21.397600 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:21.397572    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:22.939693 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:22.939662    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"]
Oct 30 03:02:22.943943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:22.943925    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:22.963649 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:22.963625    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"]
Oct 30 03:02:22.972210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:22.972189    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-ac7936-serving-cert\""
Oct 30 03:02:23.055918 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.055886    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqchm\" (UniqueName: \"kubernetes.io/projected/d440d2d2-15f6-4053-817f-19622b65948b-kube-api-access-hqchm\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.056090 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.056068    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.056162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.056145    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.056312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.056292    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d440d2d2-15f6-4053-817f-19622b65948b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.100667 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.100635    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:23.157870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.157817    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.158005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.157889    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.158005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.157973    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d440d2d2-15f6-4053-817f-19622b65948b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.158121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.158106    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqchm\" (UniqueName: \"kubernetes.io/projected/d440d2d2-15f6-4053-817f-19622b65948b-kube-api-access-hqchm\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.158668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.158648    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d440d2d2-15f6-4053-817f-19622b65948b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.160247 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.160221    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.160343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.160271    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.176489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.176463    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqchm\" (UniqueName: \"kubernetes.io/projected/d440d2d2-15f6-4053-817f-19622b65948b-kube-api-access-hqchm\") pod \"maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.232124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.232013    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:23.258332 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.258294    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:23.582792 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.582763    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"]
Oct 30 03:02:23.585770 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:02:23.585746    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd440d2d2_15f6_4053_817f_19622b65948b.slice/crio-b9478454bcead948ebeac2a4333c6aa1693db113fd83f6046ba706c2ba092fe1 WatchSource:0}: Error finding container b9478454bcead948ebeac2a4333c6aa1693db113fd83f6046ba706c2ba092fe1: Status 404 returned error can't find the container with id b9478454bcead948ebeac2a4333c6aa1693db113fd83f6046ba706c2ba092fe1
Oct 30 03:02:23.872990 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:23.872888    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp" event={"ID":"d440d2d2-15f6-4053-817f-19622b65948b","Type":"ContainerStarted","Data":"b9478454bcead948ebeac2a4333c6aa1693db113fd83f6046ba706c2ba092fe1"}
Oct 30 03:02:24.846012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:24.845974    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:24.923535 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:24.923491    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp" event={"ID":"d440d2d2-15f6-4053-817f-19622b65948b","Type":"ContainerStarted","Data":"377ae59fd8f407d0bf2011c7ac9d4c98a08f9812af297a0887f5d590012b93ec"}
Oct 30 03:02:24.923717 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:24.923644    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:24.960285 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:24.960237    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp" podStartSLOduration=2.609259384 podStartE2EDuration="2.960224233s" totalImagesPullingTime="350.964849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:24.959975984 +0000 UTC m=+1290.262154360" watchObservedRunningTime="2026-10-30 03:02:24.960224233 +0000 UTC m=+1290.262402598"
Oct 30 03:02:25.448211 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:02:25.448175    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:02:26.581985 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:26.581951    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:28.087159 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:28.087124    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:29.557434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:29.557401    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:30.640262 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.640231    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"]
Oct 30 03:02:30.646044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.646021    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.679123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.679097    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"]
Oct 30 03:02:30.685731 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.685697    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.685893 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.685767    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxwz\" (UniqueName: \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-kube-api-access-fwxwz\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.686812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.685962    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab77670-015d-4bbf-97df-75faa1cb0d18-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.686976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.686869    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.687085 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.687063    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.687168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.687150    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.687230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.687187    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.687230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.687218    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.687338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.687253    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.788286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788256    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.788433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788364    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.788433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788410    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.788433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788427    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.788589 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788445    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.788589 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788469    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.788699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788588    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.788699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788645    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxwz\" (UniqueName: \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-kube-api-access-fwxwz\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.788811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788764    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab77670-015d-4bbf-97df-75faa1cb0d18-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.788908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788859    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-credential-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.788974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788928    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-certs\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.789026 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.788996    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-data\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.789139 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.789116    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-socket\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.789421 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.789403    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab77670-015d-4bbf-97df-75faa1cb0d18-istiod-ca-cert\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.791271 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.791246    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-envoy\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.791453 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.791434    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-podinfo\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.805552 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.805525    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-token\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.805811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.805793    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxwz\" (UniqueName: \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-kube-api-access-fwxwz\") pod \"e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") " pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:30.940033 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.939951    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:02:30.961150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:30.961119    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:31.326775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:31.326707    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"]
Oct 30 03:02:31.329959 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:02:31.329921    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab77670_015d_4bbf_97df_75faa1cb0d18.slice/crio-1246cc3e14444995ac9440116dbdebe5e1a27f755f472a3b2c2c651817861cab WatchSource:0}: Error finding container 1246cc3e14444995ac9440116dbdebe5e1a27f755f472a3b2c2c651817861cab: Status 404 returned error can't find the container with id 1246cc3e14444995ac9440116dbdebe5e1a27f755f472a3b2c2c651817861cab
Oct 30 03:02:31.333918 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:31.333839    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:31.334034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:31.333969    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:31.334034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:31.334014    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:31.547129 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:31.547093    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:31.978575 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:31.978538    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" event={"ID":"8ab77670-015d-4bbf-97df-75faa1cb0d18","Type":"ContainerStarted","Data":"562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8"}
Oct 30 03:02:31.979049 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:31.978591    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" event={"ID":"8ab77670-015d-4bbf-97df-75faa1cb0d18","Type":"ContainerStarted","Data":"1246cc3e14444995ac9440116dbdebe5e1a27f755f472a3b2c2c651817861cab"}
Oct 30 03:02:32.051873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:32.045663    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" podStartSLOduration=2.045640096 podStartE2EDuration="2.045640096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:32.030233282 +0000 UTC m=+1297.332411649" watchObservedRunningTime="2026-10-30 03:02:32.045640096 +0000 UTC m=+1297.347818461"
Oct 30 03:02:32.961515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:32.961481    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:32.963250 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:32.963229    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:02:32.963358 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:32.963280    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" podUID="8ab77670-015d-4bbf-97df-75faa1cb0d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 03:02:33.444441 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:33.444407    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:33.962408 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:33.962367    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:02:33.962580 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:33.962421    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" podUID="8ab77670-015d-4bbf-97df-75faa1cb0d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 03:02:34.961707 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:34.961676    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:02:34.962119 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:34.961724    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" podUID="8ab77670-015d-4bbf-97df-75faa1cb0d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 03:02:35.008657 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:35.008625    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:35.851461 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:35.851418    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:35.961571 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:35.961535    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:02:35.961782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:35.961591    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" podUID="8ab77670-015d-4bbf-97df-75faa1cb0d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 03:02:36.846880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:36.837910    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:36.964907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:36.962749    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:02:36.964907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:36.962818    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" podUID="8ab77670-015d-4bbf-97df-75faa1cb0d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 03:02:37.968108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:37.962002    5061 patch_prober.go:29] interesting pod/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:02:37.968108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:37.962058    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" podUID="8ab77670-015d-4bbf-97df-75faa1cb0d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 03:02:38.027184 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:38.027145    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:38.148620 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:38.148587    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:38.515790 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:38.515747    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:38.931614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:38.929006    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:39.049438 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:39.049401    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:39.466528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:39.466487    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:39.537054 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:39.535155    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:39.537054 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:39.536797    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:39.562881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:39.551417    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:02:40.319324 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.319285    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"]
Oct 30 03:02:40.419865 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.419813    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.442198 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.442166    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-723ffc-openshift-default-dockercfg-5zvr6\""
Oct 30 03:02:40.477738 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.477694    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:40.552987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.552948    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"]
Oct 30 03:02:40.594447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.594347    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-certs\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.594612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.594494    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpqh8\" (UniqueName: \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-kube-api-access-lpqh8\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.594612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.594562    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-credential-socket\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.594746 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.594633    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-data\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.594746 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.594715    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-envoy\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.594874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.594745    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6dc0d03-1186-414b-b702-aa748cd029d4-istiod-ca-cert\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.594874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.594816    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-socket\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.594874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.594840    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-podinfo\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.595056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.594888    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-token\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.637394 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.637356    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:40.696666 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.696626    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpqh8\" (UniqueName: \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-kube-api-access-lpqh8\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.696898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.696705    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-credential-socket\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.696898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.696792    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-data\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.696898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.696887    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-envoy\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.697099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.696920    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6dc0d03-1186-414b-b702-aa748cd029d4-istiod-ca-cert\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.697099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.696991    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-socket\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.697099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.697016    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-podinfo\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.697099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.697047    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-token\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.697322 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.697284    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-certs\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.697823 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.697787    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-certs\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.697961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.697875    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-credential-socket\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.698174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.698149    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-socket\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.698260 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.698230    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6dc0d03-1186-414b-b702-aa748cd029d4-istiod-ca-cert\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.699008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.698983    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-data\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.700559 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.700541    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-envoy\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.704742 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.704717    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-podinfo\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.716936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.716910    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-token\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.718379 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.718348    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpqh8\" (UniqueName: \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-kube-api-access-lpqh8\") pod \"e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") " pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:40.938920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:40.937876    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:41.636962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:41.636910    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"]
Oct 30 03:02:41.671523 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:02:41.671489    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dc0d03_1186_414b_b702_aa748cd029d4.slice/crio-d281e76aff7ae6daf10c4c66460eb0c02feda28128d5281e93e20216632c4d5c WatchSource:0}: Error finding container d281e76aff7ae6daf10c4c66460eb0c02feda28128d5281e93e20216632c4d5c: Status 404 returned error can't find the container with id d281e76aff7ae6daf10c4c66460eb0c02feda28128d5281e93e20216632c4d5c
Oct 30 03:02:41.678985 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:41.677461    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:41.678985 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:41.677556    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:41.678985 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:41.677600    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:42.142950 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:42.125042    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:42.277272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:42.277186    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:42.279192 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:42.279104    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" event={"ID":"f6dc0d03-1186-414b-b702-aa748cd029d4","Type":"ContainerStarted","Data":"d281e76aff7ae6daf10c4c66460eb0c02feda28128d5281e93e20216632c4d5c"}
Oct 30 03:02:42.939870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:42.910944    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:02:43.005313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.003996    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 03:02:43.022900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.009083    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:43.023205 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.023166    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:43.268936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.247772    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 03:02:43.420873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.390238    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:43.569121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.569081    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" event={"ID":"f6dc0d03-1186-414b-b702-aa748cd029d4","Type":"ContainerStarted","Data":"8b5f98dd709733de2400bc68ef15930920f81aa10e5e6294691b46231ca1f27d"}
Oct 30 03:02:43.667872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.666332    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" podStartSLOduration=3.666311878 podStartE2EDuration="3.666311878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:43.6503134 +0000 UTC m=+1308.952491767" watchObservedRunningTime="2026-10-30 03:02:43.666311878 +0000 UTC m=+1308.968490242"
Oct 30 03:02:43.801025 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.800977    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:43.826043 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.825939    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 03:02:43.845034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.844990    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 03:02:43.846871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.846169    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:43.918875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.912924    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:02:43.918875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.913529    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 03:02:43.933961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.933918    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 03:02:43.966868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.950007    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:43.980957 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:43.980924    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:44.033489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.033442    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:44.302264 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.044023    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 03:02:44.308878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.046756    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:44.326221 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.046795    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:44.326490 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.285431    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 03:02:44.326873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.302522    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 03:02:44.327063 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.327046    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 03:02:44.347865 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.347784    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 03:02:44.348263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.348242    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 03:02:44.446877 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.440336    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 03:02:44.447436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.447409    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 03:02:44.507770 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.507726    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:44.698923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.694927    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:44.873868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.853939    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:44.902895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.876080    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 03:02:44.902895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.894958    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:44.951300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.934548    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:45.017914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.957014    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:02:45.017914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:44.981872    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:45.313895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.305940    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:45.389818 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.389773    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:45.402687 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.402644    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:45.417897 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.414931    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:45.417897 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.417048    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 03:02:45.446873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.446715    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Oct 30 03:02:45.712890 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.712735    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 03:02:45.712890 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.712835    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 03:02:45.713130 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.713076    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 03:02:45.713196 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.713156    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:02:45.713256 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.713234    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 03:02:45.713314 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.713285    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 03:02:45.713373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.713331    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 03:02:45.790881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.787179    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf"
Oct 30 03:02:45.815869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.792871    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:45.968905 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:45.968799    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:46.122875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.111697    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 03:02:46.122875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.111782    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 03:02:46.122875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.111838    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 03:02:46.122875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.111963    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 03:02:46.125534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.125509    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 03:02:46.203464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.132127    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:02:46.203464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.201275    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:46.204463 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.204439    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 03:02:46.511866 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.506570    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 03:02:46.511866 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.509389    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 03:02:46.569514 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.569484    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:46.915155 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:46.915125    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:48.106169 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.106127    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:48.248426 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.248384    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:48.267052 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.267018    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"]
Oct 30 03:02:48.272095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.272069    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:02:48.287478 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.287460    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-be9f2a59\""
Oct 30 03:02:48.287583 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.287491    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-be9f2a59-dockercfg-svgnh\""
Oct 30 03:02:48.302786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.302759    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"]
Oct 30 03:02:48.485918 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.481216    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"]
Oct 30 03:02:48.515000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.514960    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311182e9-9db5-4c05-969b-5ea5a94f29f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc\" (UID: \"311182e9-9db5-4c05-969b-5ea5a94f29f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:02:48.515465 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.515439    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6d5n\" (UniqueName: \"kubernetes.io/projected/311182e9-9db5-4c05-969b-5ea5a94f29f0-kube-api-access-c6d5n\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc\" (UID: \"311182e9-9db5-4c05-969b-5ea5a94f29f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:02:48.520979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.520957    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:02:48.545891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.542272    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"]
Oct 30 03:02:48.617438 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.617404    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6d5n\" (UniqueName: \"kubernetes.io/projected/311182e9-9db5-4c05-969b-5ea5a94f29f0-kube-api-access-c6d5n\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc\" (UID: \"311182e9-9db5-4c05-969b-5ea5a94f29f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:02:48.617582 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.617551    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-plugins-config-volume\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp\" (UID: \"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:02:48.617757 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.617734    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4p4w\" (UniqueName: \"kubernetes.io/projected/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-kube-api-access-t4p4w\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp\" (UID: \"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:02:48.617999 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.617979    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311182e9-9db5-4c05-969b-5ea5a94f29f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc\" (UID: \"311182e9-9db5-4c05-969b-5ea5a94f29f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:02:48.632873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.618897    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311182e9-9db5-4c05-969b-5ea5a94f29f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc\" (UID: \"311182e9-9db5-4c05-969b-5ea5a94f29f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:02:48.659010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.658976    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6d5n\" (UniqueName: \"kubernetes.io/projected/311182e9-9db5-4c05-969b-5ea5a94f29f0-kube-api-access-c6d5n\") pod \"payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc\" (UID: \"311182e9-9db5-4c05-969b-5ea5a94f29f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:02:48.719702 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.719664    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4p4w\" (UniqueName: \"kubernetes.io/projected/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-kube-api-access-t4p4w\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp\" (UID: \"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:02:48.720274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.720111    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-plugins-config-volume\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp\" (UID: \"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:02:48.721075 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.721052    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-plugins-config-volume\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp\" (UID: \"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:02:48.758050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.757967    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4p4w\" (UniqueName: \"kubernetes.io/projected/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-kube-api-access-t4p4w\") pod \"payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp\" (UID: \"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339\") " pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:02:48.887263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.887233    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:02:48.907353 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:48.907325    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:02:49.246889 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.245843    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"]
Oct 30 03:02:49.248531 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:02:49.248485    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311182e9_9db5_4c05_969b_5ea5a94f29f0.slice/crio-c2bcbb39a280bded20b7c0c94510adf87116ded3fc56c071b7f5d43952f87a94 WatchSource:0}: Error finding container c2bcbb39a280bded20b7c0c94510adf87116ded3fc56c071b7f5d43952f87a94: Status 404 returned error can't find the container with id c2bcbb39a280bded20b7c0c94510adf87116ded3fc56c071b7f5d43952f87a94
Oct 30 03:02:49.304890 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.304861    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"]
Oct 30 03:02:49.360165 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.360071    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:49.516210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.516174    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:49.559102 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.559014    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:02:49.559265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.559199    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:02:49.636934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.636885    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp" event={"ID":"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339","Type":"ContainerStarted","Data":"70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8"}
Oct 30 03:02:49.637100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.636978    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp" event={"ID":"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339","Type":"ContainerStarted","Data":"7caec70187a3b04f32d0fa7765ee8eabaebfbc7d1913d41877c33a1f088bcc00"}
Oct 30 03:02:49.638030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.638008    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:02:49.640212 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.640187    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc" event={"ID":"311182e9-9db5-4c05-969b-5ea5a94f29f0","Type":"ContainerStarted","Data":"26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954"}
Oct 30 03:02:49.640340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.640229    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc" event={"ID":"311182e9-9db5-4c05-969b-5ea5a94f29f0","Type":"ContainerStarted","Data":"c2bcbb39a280bded20b7c0c94510adf87116ded3fc56c071b7f5d43952f87a94"}
Oct 30 03:02:49.640340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.640249    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:02:49.679699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.679638    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp" podStartSLOduration=1.679620586 podStartE2EDuration="1.679620586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:49.675644479 +0000 UTC m=+1314.977822845" watchObservedRunningTime="2026-10-30 03:02:49.679620586 +0000 UTC m=+1314.981798952"
Oct 30 03:02:49.716512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:49.716462    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc" podStartSLOduration=1.716447375 podStartE2EDuration="1.716447375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:49.713576559 +0000 UTC m=+1315.015754937" watchObservedRunningTime="2026-10-30 03:02:49.716447375 +0000 UTC m=+1315.018625740"
Oct 30 03:02:50.538561 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:50.538530    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:50.751296 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:50.751251    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:51.034646 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:51.034608    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:51.502435 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:51.502391    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:51.646231 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:51.646202    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:52.380111 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.380082    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:52.434244 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.434204    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"]
Oct 30 03:02:52.467104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.467067    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"]
Oct 30 03:02:52.467404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.467377    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:02:52.486520 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.486474    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-723ffc\""
Oct 30 03:02:52.514522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.514488    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:52.545030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.541231    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t7nk\" (UniqueName: \"kubernetes.io/projected/1d00176e-84de-4dd2-bc2e-42d663325e2a-kube-api-access-4t7nk\") pod \"payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h\" (UID: \"1d00176e-84de-4dd2-bc2e-42d663325e2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:02:52.545030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.541342    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d00176e-84de-4dd2-bc2e-42d663325e2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h\" (UID: \"1d00176e-84de-4dd2-bc2e-42d663325e2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:02:52.649713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.646167    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t7nk\" (UniqueName: \"kubernetes.io/projected/1d00176e-84de-4dd2-bc2e-42d663325e2a-kube-api-access-4t7nk\") pod \"payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h\" (UID: \"1d00176e-84de-4dd2-bc2e-42d663325e2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:02:52.649713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.646321    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d00176e-84de-4dd2-bc2e-42d663325e2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h\" (UID: \"1d00176e-84de-4dd2-bc2e-42d663325e2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:02:52.649713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.647542    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d00176e-84de-4dd2-bc2e-42d663325e2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h\" (UID: \"1d00176e-84de-4dd2-bc2e-42d663325e2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:02:52.717582 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.717467    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t7nk\" (UniqueName: \"kubernetes.io/projected/1d00176e-84de-4dd2-bc2e-42d663325e2a-kube-api-access-4t7nk\") pod \"payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h\" (UID: \"1d00176e-84de-4dd2-bc2e-42d663325e2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:02:52.807302 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:52.807081    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:02:53.921599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:53.166200    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"]
Oct 30 03:02:53.921599 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:02:53.170003    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d00176e_84de_4dd2_bc2e_42d663325e2a.slice/crio-32e11c66300d6286658bb0cba9906aca65d8da2864054e044c090ee6a27f72eb WatchSource:0}: Error finding container 32e11c66300d6286658bb0cba9906aca65d8da2864054e044c090ee6a27f72eb: Status 404 returned error can't find the container with id 32e11c66300d6286658bb0cba9906aca65d8da2864054e044c090ee6a27f72eb
Oct 30 03:02:53.921599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:53.334537    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:53.921599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:53.742416    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h" event={"ID":"1d00176e-84de-4dd2-bc2e-42d663325e2a","Type":"ContainerStarted","Data":"d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3"}
Oct 30 03:02:53.921599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:53.742473    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:02:53.921599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:53.742490    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h" event={"ID":"1d00176e-84de-4dd2-bc2e-42d663325e2a","Type":"ContainerStarted","Data":"32e11c66300d6286658bb0cba9906aca65d8da2864054e044c090ee6a27f72eb"}
Oct 30 03:02:53.921599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:53.783830    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h" podStartSLOduration=1.783812716 podStartE2EDuration="1.783812716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:53.779243375 +0000 UTC m=+1319.081421741" watchObservedRunningTime="2026-10-30 03:02:53.783812716 +0000 UTC m=+1319.085991082"
Oct 30 03:02:54.373313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.373275    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:54.751308 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.751279    5061 generic.go:356] "Generic (PLEG): container finished" podID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerID="a1b9ee68113f7a66bdffb7771cdf89e25c483d879d6da77fcbf3c909ffb3fa62" exitCode=0
Oct 30 03:02:54.751480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.751351    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" event={"ID":"4ab1ff26-0edb-41ef-a952-ae6a17553c4c","Type":"ContainerDied","Data":"a1b9ee68113f7a66bdffb7771cdf89e25c483d879d6da77fcbf3c909ffb3fa62"}
Oct 30 03:02:54.751757 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.751740    5061 scope.go:122] "RemoveContainer" containerID="a1b9ee68113f7a66bdffb7771cdf89e25c483d879d6da77fcbf3c909ffb3fa62"
Oct 30 03:02:54.753262 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.753243    5061 generic.go:356] "Generic (PLEG): container finished" podID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerID="c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265" exitCode=0
Oct 30 03:02:54.753354 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.753320    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" event={"ID":"4a3f605d-7d6f-4e51-ada8-12357911adc7","Type":"ContainerDied","Data":"c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265"}
Oct 30 03:02:54.753354 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.753328    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:54.753467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.753401    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:54.753467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.753432    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:54.753753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.753731    5061 scope.go:122] "RemoveContainer" containerID="c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265"
Oct 30 03:02:54.755231 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.755208    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:54.755314 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.755277    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:54.755314 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.755310    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:02:54.951773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:54.951730    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:55.327948 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:55.327909    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:55.512899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:55.512793    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:55.519363 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:02:55.519277    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:02:55.958906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:55.958870    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" event={"ID":"4ab1ff26-0edb-41ef-a952-ae6a17553c4c","Type":"ContainerStarted","Data":"530e418846cf60006385ff34622d6cf71651e9c25e822c681d5acf49c531522a"}
Oct 30 03:02:55.969548 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:55.967408    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:55.976823 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:55.976786    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" event={"ID":"4a3f605d-7d6f-4e51-ada8-12357911adc7","Type":"ContainerStarted","Data":"439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9"}
Oct 30 03:02:56.176873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:56.176304    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:56.358873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:56.352098    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:56.398618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:56.398582    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:02:56.398813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:56.398792    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 03:02:56.398895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:56.398843    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 03:02:56.424574 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:56.424548    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:02:56.424932 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:56.424906    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 03:02:56.425118 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:56.425094    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 03:02:56.634939 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:56.634686    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:57.381127 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:57.381093    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:57.398875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:57.398816    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 03:02:57.399003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:57.398885    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 03:02:57.427877 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:57.426816    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 03:02:57.427877 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:57.426904    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 03:02:57.978885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:57.978162    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:58.083888 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.083613    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"]
Oct 30 03:02:58.087265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.087239    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:02:58.091460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.091430    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"]
Oct 30 03:02:58.095117 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.095093    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"]
Oct 30 03:02:58.098569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.098551    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:02:58.104484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.104457    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-ae1dd9\""
Oct 30 03:02:58.107689 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.107673    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:58.116231 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.116101    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-ae1dd9-dockercfg-9jxwf\""
Oct 30 03:02:58.117967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.117946    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"]
Oct 30 03:02:58.414894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.408974    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm\" (UID: \"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:02:58.414894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.409041    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776x5\" (UniqueName: \"kubernetes.io/projected/d29cfa43-cc24-4c84-a55d-12401012d345-kube-api-access-776x5\") pod \"payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9\" (UID: \"d29cfa43-cc24-4c84-a55d-12401012d345\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:02:58.414894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.409122    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfw7m\" (UniqueName: \"kubernetes.io/projected/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-kube-api-access-hfw7m\") pod \"payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm\" (UID: \"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:02:58.414894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.409376    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d29cfa43-cc24-4c84-a55d-12401012d345-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9\" (UID: \"d29cfa43-cc24-4c84-a55d-12401012d345\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:02:58.414894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.410192    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 03:02:58.414894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.410245    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 03:02:58.512030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.477783    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 03:02:58.512030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.477876    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 03:02:58.512030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.511040    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm\" (UID: \"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:02:58.512030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.511102    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776x5\" (UniqueName: \"kubernetes.io/projected/d29cfa43-cc24-4c84-a55d-12401012d345-kube-api-access-776x5\") pod \"payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9\" (UID: \"d29cfa43-cc24-4c84-a55d-12401012d345\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:02:58.512030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.511169    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfw7m\" (UniqueName: \"kubernetes.io/projected/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-kube-api-access-hfw7m\") pod \"payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm\" (UID: \"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:02:58.512030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.511383    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d29cfa43-cc24-4c84-a55d-12401012d345-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9\" (UID: \"d29cfa43-cc24-4c84-a55d-12401012d345\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:02:58.512584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.512474    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d29cfa43-cc24-4c84-a55d-12401012d345-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9\" (UID: \"d29cfa43-cc24-4c84-a55d-12401012d345\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:02:58.513287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.513261    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm\" (UID: \"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:02:58.626872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.622717    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfw7m\" (UniqueName: \"kubernetes.io/projected/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-kube-api-access-hfw7m\") pod \"payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm\" (UID: \"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:02:58.695923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.683775    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-776x5\" (UniqueName: \"kubernetes.io/projected/d29cfa43-cc24-4c84-a55d-12401012d345-kube-api-access-776x5\") pod \"payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9\" (UID: \"d29cfa43-cc24-4c84-a55d-12401012d345\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:02:58.718425 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.718390    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:02:58.744747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:58.744712    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:02:59.271585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:59.270607    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"]
Oct 30 03:02:59.377955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:59.377124    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"]
Oct 30 03:02:59.399609 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:59.399581    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 03:02:59.399727 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:59.399626    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 03:02:59.423393 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:59.423292    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 03:02:59.423393 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:59.423339    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 03:02:59.797934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:59.797898    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:59.970991 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:02:59.970898    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:00.103875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.092216    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9" event={"ID":"d29cfa43-cc24-4c84-a55d-12401012d345","Type":"ContainerStarted","Data":"1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4"}
Oct 30 03:03:00.103875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.092287    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9" event={"ID":"d29cfa43-cc24-4c84-a55d-12401012d345","Type":"ContainerStarted","Data":"616ee5a30a24dce5012443ff1d2ef1042a271836c0bf0649b327f7eadea0ceb4"}
Oct 30 03:03:00.103875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.093299    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:03:00.127868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.120738    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm" event={"ID":"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464","Type":"ContainerStarted","Data":"c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582"}
Oct 30 03:03:00.127868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.120811    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm" event={"ID":"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464","Type":"ContainerStarted","Data":"e487eb0c09e7637d301c0c547c82a645c437e3f7411a3a3a6300dda811cb5d86"}
Oct 30 03:03:00.127868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.125928    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:03:00.127868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.126181    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:00.162093 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.162027    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9" podStartSLOduration=2.162008609 podStartE2EDuration="2.162008609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:00.1570626 +0000 UTC m=+1325.459240966" watchObservedRunningTime="2026-10-30 03:03:00.162008609 +0000 UTC m=+1325.464186974"
Oct 30 03:03:00.220181 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.220100    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm" podStartSLOduration=3.220077325 podStartE2EDuration="3.220077325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:00.218296039 +0000 UTC m=+1325.520474407" watchObservedRunningTime="2026-10-30 03:03:00.220077325 +0000 UTC m=+1325.522255720"
Oct 30 03:03:00.418894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.418838    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 03:03:00.420067 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.420024    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 03:03:00.432009 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.431981    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 03:03:00.432537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.432503    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 03:03:00.675735 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:00.675638    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:03:01.397954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:01.397920    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 03:03:01.398104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:01.397991    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 03:03:01.592350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:01.592317    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:03:01.592873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:01.592687    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:03:01.593773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:01.593753    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:03:01.673286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:01.673200    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:03:01.854368 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:01.854333    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:01.991260 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:01.991179    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"]
Oct 30 03:03:01.994901 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:01.994879    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:03:02.003134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.003112    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:02.006333 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.006307    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmvn7\" (UniqueName: \"kubernetes.io/projected/dff57095-f4c4-416b-885e-ede0cd4087f3-kube-api-access-tmvn7\") pod \"payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld\" (UID: \"dff57095-f4c4-416b-885e-ede0cd4087f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:03:02.006436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.006365    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff57095-f4c4-416b-885e-ede0cd4087f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld\" (UID: \"dff57095-f4c4-416b-885e-ede0cd4087f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:03:02.013578 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.013559    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-9d3552\""
Oct 30 03:03:02.015498 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.015479    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"]
Oct 30 03:03:02.064081 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.064048    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"]
Oct 30 03:03:02.068920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.068881    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:03:02.077006 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.076981    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-9d3552-dockercfg-hv795\""
Oct 30 03:03:02.098647 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.098609    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"]
Oct 30 03:03:02.156447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.152157    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91673351-2242-4930-be49-6482e35194e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz\" (UID: \"91673351-2242-4930-be49-6482e35194e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:03:02.156447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.152305    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmvn7\" (UniqueName: \"kubernetes.io/projected/dff57095-f4c4-416b-885e-ede0cd4087f3-kube-api-access-tmvn7\") pod \"payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld\" (UID: \"dff57095-f4c4-416b-885e-ede0cd4087f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:03:02.156447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.152368    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff57095-f4c4-416b-885e-ede0cd4087f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld\" (UID: \"dff57095-f4c4-416b-885e-ede0cd4087f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:03:02.156447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.152495    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbz5\" (UniqueName: \"kubernetes.io/projected/91673351-2242-4930-be49-6482e35194e8-kube-api-access-dtbz5\") pod \"payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz\" (UID: \"91673351-2242-4930-be49-6482e35194e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:03:02.161108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.156481    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff57095-f4c4-416b-885e-ede0cd4087f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld\" (UID: \"dff57095-f4c4-416b-885e-ede0cd4087f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:03:02.203916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.187401    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmvn7\" (UniqueName: \"kubernetes.io/projected/dff57095-f4c4-416b-885e-ede0cd4087f3-kube-api-access-tmvn7\") pod \"payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld\" (UID: \"dff57095-f4c4-416b-885e-ede0cd4087f3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:03:02.242545 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.240727    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:02.256086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.256041    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbz5\" (UniqueName: \"kubernetes.io/projected/91673351-2242-4930-be49-6482e35194e8-kube-api-access-dtbz5\") pod \"payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz\" (UID: \"91673351-2242-4930-be49-6482e35194e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:03:02.256542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.256520    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91673351-2242-4930-be49-6482e35194e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz\" (UID: \"91673351-2242-4930-be49-6482e35194e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:03:02.275908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.257708    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91673351-2242-4930-be49-6482e35194e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz\" (UID: \"91673351-2242-4930-be49-6482e35194e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:03:02.295896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.295542    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbz5\" (UniqueName: \"kubernetes.io/projected/91673351-2242-4930-be49-6482e35194e8-kube-api-access-dtbz5\") pod \"payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz\" (UID: \"91673351-2242-4930-be49-6482e35194e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:03:02.310065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.310035    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:03:02.408322 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.404981    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:03:02.408322 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.405713    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:03:02.408322 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.406390    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:03:02.456887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.453931    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:03:02.624541 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.624400    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"]
Oct 30 03:03:02.829797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:02.828830    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"]
Oct 30 03:03:03.245338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:03.245237    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz" event={"ID":"91673351-2242-4930-be49-6482e35194e8","Type":"ContainerStarted","Data":"a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7"}
Oct 30 03:03:03.245338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:03.245301    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:03:03.245338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:03.245324    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz" event={"ID":"91673351-2242-4930-be49-6482e35194e8","Type":"ContainerStarted","Data":"77d3707dcce54c4f18df66c88c3588a2ad9ecfa207f7191ad02cd95ff3c3e508"}
Oct 30 03:03:03.246795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:03.246765    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld" event={"ID":"dff57095-f4c4-416b-885e-ede0cd4087f3","Type":"ContainerStarted","Data":"13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680"}
Oct 30 03:03:03.246954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:03.246819    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld" event={"ID":"dff57095-f4c4-416b-885e-ede0cd4087f3","Type":"ContainerStarted","Data":"bccd3e90ea761fb0c2c17d9781518d84d5e4e0df8e9adb38114a6b5b36b8582c"}
Oct 30 03:03:03.246954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:03.246892    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:03:03.284501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:03.284460    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz" podStartSLOduration=1.284447577 podStartE2EDuration="1.284447577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:03.279458378 +0000 UTC m=+1328.581636743" watchObservedRunningTime="2026-10-30 03:03:03.284447577 +0000 UTC m=+1328.586625983"
Oct 30 03:03:03.317059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:03.317019    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld" podStartSLOduration=2.317006513 podStartE2EDuration="2.317006513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:03.313952568 +0000 UTC m=+1328.616130930" watchObservedRunningTime="2026-10-30 03:03:03.317006513 +0000 UTC m=+1328.619184877"
Oct 30 03:03:04.116692 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:04.116661    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:04.755919 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:04.755891    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:03:05.943334 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:05.943303    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:07.900210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:07.900175    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:09.572624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:09.572589    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:10.918085 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:10.918018    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:11.175592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:11.175514    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:11.379823 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:11.379787    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:11.580046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:11.580013    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:12.242139 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:12.242110    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:03:12.242551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:12.242176    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:03:13.310877 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:13.310811    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:14.043898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:14.043818    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:14.258390 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:14.258363    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:03:14.258538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:14.258429    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:03:14.600230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:14.600198    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:15.106234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:15.106194    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:15.490359 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:15.490274    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:15.611563 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:15.611522    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:15.741840 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:15.741742    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:17.610405 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:17.610372    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:18.178255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:18.178220    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:18.315358 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:18.315329    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:19.493429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:19.493392    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:19.968541 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:19.968504    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:20.119431 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.119394    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:20.126973 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.126945    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"]
Oct 30 03:03:20.130614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.130598    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:03:20.145342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.145314    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d7fde0-dockercfg-c7t4c\""
Oct 30 03:03:20.151024 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.150998    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"]
Oct 30 03:03:20.155827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.155806    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d7fde0\""
Oct 30 03:03:20.209880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.209805    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c421f7-50d6-4401-941d-542df467aa4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj\" (UID: \"d6c421f7-50d6-4401-941d-542df467aa4f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:03:20.210071 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.209907    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg4gb\" (UniqueName: \"kubernetes.io/projected/d6c421f7-50d6-4401-941d-542df467aa4f-kube-api-access-pg4gb\") pod \"payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj\" (UID: \"d6c421f7-50d6-4401-941d-542df467aa4f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:03:20.211368 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.211341    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"]
Oct 30 03:03:20.360316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.360280    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"]
Oct 30 03:03:20.360643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.360624    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:03:20.369089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.369059    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c421f7-50d6-4401-941d-542df467aa4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj\" (UID: \"d6c421f7-50d6-4401-941d-542df467aa4f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:03:20.381553 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.370467    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c421f7-50d6-4401-941d-542df467aa4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj\" (UID: \"d6c421f7-50d6-4401-941d-542df467aa4f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:03:20.381796 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.381760    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg4gb\" (UniqueName: \"kubernetes.io/projected/d6c421f7-50d6-4401-941d-542df467aa4f-kube-api-access-pg4gb\") pod \"payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj\" (UID: \"d6c421f7-50d6-4401-941d-542df467aa4f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:03:20.413083 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.413044    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg4gb\" (UniqueName: \"kubernetes.io/projected/d6c421f7-50d6-4401-941d-542df467aa4f-kube-api-access-pg4gb\") pod \"payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj\" (UID: \"d6c421f7-50d6-4401-941d-542df467aa4f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:03:20.446648 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.446614    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:03:20.484495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.483838    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d8f59d-ff26-4c89-8c35-896b6bda6b75-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj\" (UID: \"59d8f59d-ff26-4c89-8c35-896b6bda6b75\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:03:20.484495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.484172    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dspc5\" (UniqueName: \"kubernetes.io/projected/59d8f59d-ff26-4c89-8c35-896b6bda6b75-kube-api-access-dspc5\") pod \"payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj\" (UID: \"59d8f59d-ff26-4c89-8c35-896b6bda6b75\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:03:20.596876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.588915    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d8f59d-ff26-4c89-8c35-896b6bda6b75-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj\" (UID: \"59d8f59d-ff26-4c89-8c35-896b6bda6b75\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:03:20.596876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.588976    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d8f59d-ff26-4c89-8c35-896b6bda6b75-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj\" (UID: \"59d8f59d-ff26-4c89-8c35-896b6bda6b75\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:03:20.596876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.589293    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dspc5\" (UniqueName: \"kubernetes.io/projected/59d8f59d-ff26-4c89-8c35-896b6bda6b75-kube-api-access-dspc5\") pod \"payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj\" (UID: \"59d8f59d-ff26-4c89-8c35-896b6bda6b75\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:03:20.643131 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.643051    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dspc5\" (UniqueName: \"kubernetes.io/projected/59d8f59d-ff26-4c89-8c35-896b6bda6b75-kube-api-access-dspc5\") pod \"payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj\" (UID: \"59d8f59d-ff26-4c89-8c35-896b6bda6b75\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:03:20.685824 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.685792    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:03:20.771643 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:03:20.771610    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c421f7_50d6_4401_941d_542df467aa4f.slice/crio-66e6dbcd95f009ad50301a97d82de0e1884701905ca0cde72ad3da6029ea9a9f WatchSource:0}: Error finding container 66e6dbcd95f009ad50301a97d82de0e1884701905ca0cde72ad3da6029ea9a9f: Status 404 returned error can't find the container with id 66e6dbcd95f009ad50301a97d82de0e1884701905ca0cde72ad3da6029ea9a9f
Oct 30 03:03:20.771817 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:20.771790    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"]
Oct 30 03:03:21.011965 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:21.011941    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"]
Oct 30 03:03:21.013680 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:03:21.013655    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d8f59d_ff26_4c89_8c35_896b6bda6b75.slice/crio-ccc889273b2717ab0e2ee985b33da68bd434900ee0bffa44165875199f691308 WatchSource:0}: Error finding container ccc889273b2717ab0e2ee985b33da68bd434900ee0bffa44165875199f691308: Status 404 returned error can't find the container with id ccc889273b2717ab0e2ee985b33da68bd434900ee0bffa44165875199f691308
Oct 30 03:03:21.447921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:21.447874    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj" event={"ID":"59d8f59d-ff26-4c89-8c35-896b6bda6b75","Type":"ContainerStarted","Data":"bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba"}
Oct 30 03:03:21.448082 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:21.447935    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj" event={"ID":"59d8f59d-ff26-4c89-8c35-896b6bda6b75","Type":"ContainerStarted","Data":"ccc889273b2717ab0e2ee985b33da68bd434900ee0bffa44165875199f691308"}
Oct 30 03:03:21.448082 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:21.447957    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:03:21.449509 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:21.449484    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj" event={"ID":"d6c421f7-50d6-4401-941d-542df467aa4f","Type":"ContainerStarted","Data":"a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5"}
Oct 30 03:03:21.449627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:21.449523    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj" event={"ID":"d6c421f7-50d6-4401-941d-542df467aa4f","Type":"ContainerStarted","Data":"66e6dbcd95f009ad50301a97d82de0e1884701905ca0cde72ad3da6029ea9a9f"}
Oct 30 03:03:21.449627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:21.449553    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:03:21.491188 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:21.491145    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj" podStartSLOduration=1.491132161 podStartE2EDuration="1.491132161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:21.486545274 +0000 UTC m=+1346.788723638" watchObservedRunningTime="2026-10-30 03:03:21.491132161 +0000 UTC m=+1346.793310525"
Oct 30 03:03:21.531351 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:21.531308    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj" podStartSLOduration=1.531295968 podStartE2EDuration="1.531295968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:21.527506633 +0000 UTC m=+1346.829684998" watchObservedRunningTime="2026-10-30 03:03:21.531295968 +0000 UTC m=+1346.833474333"
Oct 30 03:03:21.705101 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:21.705010    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:23.250359 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:23.250322    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:23.368229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:23.367603    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:25.314123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:25.314091    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:25.458719 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:03:25.458681    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:03:26.885497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:26.885458    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:28.247396 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.247359    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"]
Oct 30 03:03:28.253069 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.253048    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.270625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.270602    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"]
Oct 30 03:03:28.278814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.276215    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-9d3552-serving-cert\""
Oct 30 03:03:28.315451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.315418    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.315451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.315463    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.315665 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.315624    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f6ecbf-c78d-4634-844a-7652c4c75658-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.315807 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.315791    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wk55\" (UniqueName: \"kubernetes.io/projected/30f6ecbf-c78d-4634-844a-7652c4c75658-kube-api-access-5wk55\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.364099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.364075    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:28.416692 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.416665    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.416836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.416698    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.416836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.416812    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f6ecbf-c78d-4634-844a-7652c4c75658-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.417025 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.417005    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wk55\" (UniqueName: \"kubernetes.io/projected/30f6ecbf-c78d-4634-844a-7652c4c75658-kube-api-access-5wk55\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.417645 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.417619    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f6ecbf-c78d-4634-844a-7652c4c75658-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.419544 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.419516    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.419737 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.419718    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.435139 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.435116    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wk55\" (UniqueName: \"kubernetes.io/projected/30f6ecbf-c78d-4634-844a-7652c4c75658-kube-api-access-5wk55\") pod \"maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.510953 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.510835    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:28.568452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.568411    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:28.757320 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.757283    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:28.900961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:28.900931    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"]
Oct 30 03:03:28.904902 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:03:28.904871    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f6ecbf_c78d_4634_844a_7652c4c75658.slice/crio-098754d3e8a6638bcdd1e6a2de1283850b2d69ace9cf1b8fec3974c24dd92c97 WatchSource:0}: Error finding container 098754d3e8a6638bcdd1e6a2de1283850b2d69ace9cf1b8fec3974c24dd92c97: Status 404 returned error can't find the container with id 098754d3e8a6638bcdd1e6a2de1283850b2d69ace9cf1b8fec3974c24dd92c97
Oct 30 03:03:29.552058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:29.552015    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b" event={"ID":"30f6ecbf-c78d-4634-844a-7652c4c75658","Type":"ContainerStarted","Data":"dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb"}
Oct 30 03:03:29.554032 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:29.554001    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b" event={"ID":"30f6ecbf-c78d-4634-844a-7652c4c75658","Type":"ContainerStarted","Data":"098754d3e8a6638bcdd1e6a2de1283850b2d69ace9cf1b8fec3974c24dd92c97"}
Oct 30 03:03:29.554322 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:29.554294    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:29.592758 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:29.592638    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b" podStartSLOduration=1.193365876 podStartE2EDuration="1.592619277s" totalImagesPullingTime="399.253401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:29.590770431 +0000 UTC m=+1354.892948796" watchObservedRunningTime="2026-10-30 03:03:29.592619277 +0000 UTC m=+1354.894797669"
Oct 30 03:03:30.442572 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:30.442531    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:32.072749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:32.072713    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:32.460444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:32.460354    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:03:32.460444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:32.460430    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:03:32.622410 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:32.622380    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:33.607806 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:33.607773    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:35.228893 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:35.228830    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:36.573717 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:36.573690    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:03:36.783035 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:36.782985    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:38.298751 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:38.298714    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:38.412328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:38.412289    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:40.104515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:40.104482    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:42.001236 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:42.001187    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:42.373438 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:42.373400    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:43.264030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.263993    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"]
Oct 30 03:03:43.269879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.269835    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.300304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.300268    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"]
Oct 30 03:03:43.310192 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.310158    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7ffb1886-openshift-default-dockercfg-bdn5p\""
Oct 30 03:03:43.390966 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.390923    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-certs\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.391173 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.391023    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-credential-socket\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.391173 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.391082    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-istio-token\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.391299 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.391187    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059a5e38-5767-417c-93d4-54b7af112f2f-istiod-ca-cert\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.391299 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.391226    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-data\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.391299 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.391255    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059a5e38-5767-417c-93d4-54b7af112f2f-istio-podinfo\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.391455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.391294    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzxbr\" (UniqueName: \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-kube-api-access-qzxbr\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.391455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.391341    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-socket\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.391455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.391409    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-envoy\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.396442 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.396408    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:43.501867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.493520    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-credential-socket\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.501867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.493596    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-istio-token\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.501867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.493711    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059a5e38-5767-417c-93d4-54b7af112f2f-istiod-ca-cert\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.501867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.493754    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-data\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.501867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.493801    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059a5e38-5767-417c-93d4-54b7af112f2f-istio-podinfo\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.501867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.493863    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxbr\" (UniqueName: \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-kube-api-access-qzxbr\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.501867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.493907    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-socket\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.501867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.493951    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-envoy\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.501867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.494201    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-certs\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.501867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.494740    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-certs\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.501867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.495011    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-credential-socket\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.503474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.502741    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059a5e38-5767-417c-93d4-54b7af112f2f-istiod-ca-cert\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.503474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.503049    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-data\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.505346 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.505320    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-socket\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.528952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.527764    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059a5e38-5767-417c-93d4-54b7af112f2f-istio-podinfo\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.532124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.532098    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-istio-token\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.533936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.533890    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxbr\" (UniqueName: \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-kube-api-access-qzxbr\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.534172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.534137    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-envoy\") pod \"e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") " pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.587485 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.587459    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:43.674937 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.672665    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:43.942764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.942730    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"]
Oct 30 03:03:43.969946 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.969909    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:03:43.970070 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.969996    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:03:43.970070 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:43.970044    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:03:44.750448 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:44.750413    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" event={"ID":"059a5e38-5767-417c-93d4-54b7af112f2f","Type":"ContainerStarted","Data":"7cae930294489af94369add82e07fa846162e2e3089ec58e3a08163a064352ee"}
Oct 30 03:03:44.750965 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:44.750927    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" event={"ID":"059a5e38-5767-417c-93d4-54b7af112f2f","Type":"ContainerStarted","Data":"ad039b6ee1fd2f2518408ac3859d96dedb17f6b9aa8a03bcbaa672df8986bbdd"}
Oct 30 03:03:44.814969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:44.814918    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" podStartSLOduration=1.8149044509999999 podStartE2EDuration="1.814904451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:44.800335534 +0000 UTC m=+1370.102513897" watchObservedRunningTime="2026-10-30 03:03:44.814904451 +0000 UTC m=+1370.117082815"
Oct 30 03:03:45.182098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:45.182054    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:45.587779 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:45.587747    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:45.594092 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:45.594065    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:45.759249 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:45.759221    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:45.760254 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:45.760235    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:03:46.804052 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:46.804007    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:47.083696 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:47.083556    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:47.645357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:47.645321    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:47.775610 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:47.775566    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:47.974053 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:47.973964    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:48.105882 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:48.105815    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:50.309162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:50.309091    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:50.616538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:50.616449    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:50.814401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:50.814363    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:50.954984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:50.954899    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"]
Oct 30 03:03:50.963987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:50.962711    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:03:50.987805 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:50.987756    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"]
Oct 30 03:03:50.989738 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:50.989707    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:50.992029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:50.991996    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7ffb1886-dockercfg-lqbr5\""
Oct 30 03:03:51.005784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.005608    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7ffb1886\""
Oct 30 03:03:51.040265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.036041    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw9r4\" (UniqueName: \"kubernetes.io/projected/1848202f-6bd1-4c94-9e2b-b8627325c908-kube-api-access-xw9r4\") pod \"payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4\" (UID: \"1848202f-6bd1-4c94-9e2b-b8627325c908\") " pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:03:51.040265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.036316    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1848202f-6bd1-4c94-9e2b-b8627325c908-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4\" (UID: \"1848202f-6bd1-4c94-9e2b-b8627325c908\") " pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:03:51.137822 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.137788    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1848202f-6bd1-4c94-9e2b-b8627325c908-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4\" (UID: \"1848202f-6bd1-4c94-9e2b-b8627325c908\") " pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:03:51.138060 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.137949    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw9r4\" (UniqueName: \"kubernetes.io/projected/1848202f-6bd1-4c94-9e2b-b8627325c908-kube-api-access-xw9r4\") pod \"payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4\" (UID: \"1848202f-6bd1-4c94-9e2b-b8627325c908\") " pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:03:51.138756 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.138728    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1848202f-6bd1-4c94-9e2b-b8627325c908-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4\" (UID: \"1848202f-6bd1-4c94-9e2b-b8627325c908\") " pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:03:51.159315 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.159286    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw9r4\" (UniqueName: \"kubernetes.io/projected/1848202f-6bd1-4c94-9e2b-b8627325c908-kube-api-access-xw9r4\") pod \"payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4\" (UID: \"1848202f-6bd1-4c94-9e2b-b8627325c908\") " pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:03:51.297362 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.297327    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:03:51.351871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.346698    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:51.699744 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.699678    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"]
Oct 30 03:03:51.853134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.846525    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4" event={"ID":"1848202f-6bd1-4c94-9e2b-b8627325c908","Type":"ContainerStarted","Data":"3702b43182b2e6faed20387919dd26635768bc2f7bcbc9bff8e26c57f0d1f58f"}
Oct 30 03:03:51.975938 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:51.975203    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:52.128596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:52.128405    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:52.857442 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:52.857397    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4" event={"ID":"1848202f-6bd1-4c94-9e2b-b8627325c908","Type":"ContainerStarted","Data":"3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10"}
Oct 30 03:03:52.857947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:52.857532    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:03:52.900876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:52.900802    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4" podStartSLOduration=2.900783942 podStartE2EDuration="2.900783942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:52.896965848 +0000 UTC m=+1378.199144210" watchObservedRunningTime="2026-10-30 03:03:52.900783942 +0000 UTC m=+1378.202962309"
Oct 30 03:03:53.243427 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:53.243336    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:54.881882 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:54.881831    5061 generic.go:356] "Generic (PLEG): container finished" podID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerID="8b5f98dd709733de2400bc68ef15930920f81aa10e5e6294691b46231ca1f27d" exitCode=0
Oct 30 03:03:54.882313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:54.881920    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" event={"ID":"f6dc0d03-1186-414b-b702-aa748cd029d4","Type":"ContainerDied","Data":"8b5f98dd709733de2400bc68ef15930920f81aa10e5e6294691b46231ca1f27d"}
Oct 30 03:03:54.882407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:54.882389    5061 scope.go:122] "RemoveContainer" containerID="8b5f98dd709733de2400bc68ef15930920f81aa10e5e6294691b46231ca1f27d"
Oct 30 03:03:54.884393 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:54.884365    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:03:54.884488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:54.884452    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:03:54.884527 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:54.884493    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:03:55.343543 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:55.343509    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:55.468624 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:03:55.468585    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:03:55.891366 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:55.891330    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" event={"ID":"f6dc0d03-1186-414b-b702-aa748cd029d4","Type":"ContainerStarted","Data":"e474a967146a61644872461d96251e2511c2580a14297bc92434e5f4e065e0e4"}
Oct 30 03:03:55.951210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:55.951177    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:03:55.952002 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:55.951970    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 03:03:55.952148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:55.952029    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 03:03:56.947711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:56.946019    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 03:03:56.947711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:56.946084    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 03:03:57.143505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:57.143468    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:57.285103 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:57.285027    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:57.418800 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:57.418764    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:57.943881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:57.943505    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 03:03:57.943881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:57.943564    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 03:03:58.944307 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:58.944261    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 03:03:58.944786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:58.944745    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 03:03:58.963330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:58.963304    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:59.049756 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.049717    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"]
Oct 30 03:03:59.054995 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.054967    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"]
Oct 30 03:03:59.055157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.055082    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:03:59.067971 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.067943    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8e2727\""
Oct 30 03:03:59.078520 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.078489    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8e2727-dockercfg-f8tr7\""
Oct 30 03:03:59.100178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.100150    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:59.192496 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.192453    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt\" (UID: \"7b5fe7e6-dd7b-4386-8504-ea4507d6541e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:03:59.209925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.192982    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clmng\" (UniqueName: \"kubernetes.io/projected/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-kube-api-access-clmng\") pod \"payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt\" (UID: \"7b5fe7e6-dd7b-4386-8504-ea4507d6541e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:03:59.458909 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.458833    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clmng\" (UniqueName: \"kubernetes.io/projected/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-kube-api-access-clmng\") pod \"payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt\" (UID: \"7b5fe7e6-dd7b-4386-8504-ea4507d6541e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:03:59.503860 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.501510    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt\" (UID: \"7b5fe7e6-dd7b-4386-8504-ea4507d6541e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:03:59.503860 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.502642    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt\" (UID: \"7b5fe7e6-dd7b-4386-8504-ea4507d6541e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:03:59.507029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.506997    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clmng\" (UniqueName: \"kubernetes.io/projected/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-kube-api-access-clmng\") pod \"payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt\" (UID: \"7b5fe7e6-dd7b-4386-8504-ea4507d6541e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:03:59.528942 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.528912    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"]
Oct 30 03:03:59.543522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.543495    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"]
Oct 30 03:03:59.547770 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.546661    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:03:59.603020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.602983    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4\" (UID: \"f8bb1f50-839d-44e1-9d5a-9b0401f324c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:03:59.603300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.603277    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grhw5\" (UniqueName: \"kubernetes.io/projected/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-kube-api-access-grhw5\") pod \"payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4\" (UID: \"f8bb1f50-839d-44e1-9d5a-9b0401f324c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:03:59.674115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.674074    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:03:59.695019 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.694985    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:03:59.705532 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.705500    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grhw5\" (UniqueName: \"kubernetes.io/projected/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-kube-api-access-grhw5\") pod \"payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4\" (UID: \"f8bb1f50-839d-44e1-9d5a-9b0401f324c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:03:59.705704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.705682    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4\" (UID: \"f8bb1f50-839d-44e1-9d5a-9b0401f324c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:03:59.706618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.706584    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4\" (UID: \"f8bb1f50-839d-44e1-9d5a-9b0401f324c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:03:59.752683 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.752641    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grhw5\" (UniqueName: \"kubernetes.io/projected/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-kube-api-access-grhw5\") pod \"payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4\" (UID: \"f8bb1f50-839d-44e1-9d5a-9b0401f324c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:03:59.762128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.761212    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:59.883504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.883458    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:03:59.961642 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.961423    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 03:03:59.961642 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:03:59.961534    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 03:04:00.058120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:00.058079    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"]
Oct 30 03:04:00.258404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:00.258371    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"]
Oct 30 03:04:00.259729 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:04:00.259699    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bb1f50_839d_44e1_9d5a_9b0401f324c0.slice/crio-ee638e83e28c86ee0ad0ba415af3be93745c30314c32a65fb9f2ad418bd4fe43 WatchSource:0}: Error finding container ee638e83e28c86ee0ad0ba415af3be93745c30314c32a65fb9f2ad418bd4fe43: Status 404 returned error can't find the container with id ee638e83e28c86ee0ad0ba415af3be93745c30314c32a65fb9f2ad418bd4fe43
Oct 30 03:04:00.942880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:00.942178    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 03:04:00.942880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:00.942238    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 03:04:00.978641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:00.978608    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt" event={"ID":"7b5fe7e6-dd7b-4386-8504-ea4507d6541e","Type":"ContainerStarted","Data":"63eb29e7ca2230f607870f6ad125877f6b91535386ae4c8f612eef5f9fc8072f"}
Oct 30 03:04:00.979114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:00.978673    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt" event={"ID":"7b5fe7e6-dd7b-4386-8504-ea4507d6541e","Type":"ContainerStarted","Data":"3c2890fff0c60634bf8d162cda855219708115a1ba4ea58dd3ca3b416c98f65c"}
Oct 30 03:04:00.979714 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:00.979688    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:04:00.981500 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:00.981458    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4" event={"ID":"f8bb1f50-839d-44e1-9d5a-9b0401f324c0","Type":"ContainerStarted","Data":"faa01350fc7e59763c1f9d05cd6b9af30e28b2b4f58710ad2bd6ed3d875143cc"}
Oct 30 03:04:00.981607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:00.981503    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4" event={"ID":"f8bb1f50-839d-44e1-9d5a-9b0401f324c0","Type":"ContainerStarted","Data":"ee638e83e28c86ee0ad0ba415af3be93745c30314c32a65fb9f2ad418bd4fe43"}
Oct 30 03:04:00.981607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:00.981528    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:04:01.038868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:01.018907    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt" podStartSLOduration=2.018890997 podStartE2EDuration="2.018890997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:01.0151452 +0000 UTC m=+1386.317323562" watchObservedRunningTime="2026-10-30 03:04:01.018890997 +0000 UTC m=+1386.321069363"
Oct 30 03:04:01.067472 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:01.067424    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4" podStartSLOduration=2.067409846 podStartE2EDuration="2.067409846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:01.065095281 +0000 UTC m=+1386.367273646" watchObservedRunningTime="2026-10-30 03:04:01.067409846 +0000 UTC m=+1386.369588212"
Oct 30 03:04:01.814227 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:01.814192    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:01.945929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:01.945890    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:04:01.947522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:01.947495    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:04:02.403718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:02.403683    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:03.816172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:03.816136    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:03.881130 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:03.881087    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:04:04.059268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:04.059222    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:04.462821 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:04.462771    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:04.588866 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:04.588815    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:06.000474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:06.000365    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:06.563216 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:06.563186    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:07.973460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:07.973426    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:08.697800 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:08.697769    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:08.898922 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:08.898884    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:09.966555 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:09.966525    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:10.439338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.439299    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:10.589543 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.589514    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"]
Oct 30 03:04:10.600866 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.594646    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:10.605084 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.603486    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:04:10.614612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.614580    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"]
Oct 30 03:04:10.618581 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.618549    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-44a8fd\""
Oct 30 03:04:10.618696 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.618669    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-44a8fd-dockercfg-457nx\""
Oct 30 03:04:10.812494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.812452    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbtr\" (UniqueName: \"kubernetes.io/projected/82ca6e05-6e01-44e5-913a-c285f53af294-kube-api-access-gnbtr\") pod \"payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd\" (UID: \"82ca6e05-6e01-44e5-913a-c285f53af294\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:04:10.812763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.812746    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ca6e05-6e01-44e5-913a-c285f53af294-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd\" (UID: \"82ca6e05-6e01-44e5-913a-c285f53af294\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:04:10.932839 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.930662    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbtr\" (UniqueName: \"kubernetes.io/projected/82ca6e05-6e01-44e5-913a-c285f53af294-kube-api-access-gnbtr\") pod \"payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd\" (UID: \"82ca6e05-6e01-44e5-913a-c285f53af294\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:04:10.933924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.933898    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ca6e05-6e01-44e5-913a-c285f53af294-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd\" (UID: \"82ca6e05-6e01-44e5-913a-c285f53af294\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:04:10.935388 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.934961    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ca6e05-6e01-44e5-913a-c285f53af294-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd\" (UID: \"82ca6e05-6e01-44e5-913a-c285f53af294\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:04:10.940276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.940255    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"]
Oct 30 03:04:10.951557 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.951530    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbtr\" (UniqueName: \"kubernetes.io/projected/82ca6e05-6e01-44e5-913a-c285f53af294-kube-api-access-gnbtr\") pod \"payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd\" (UID: \"82ca6e05-6e01-44e5-913a-c285f53af294\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:04:10.959194 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.959171    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:04:10.962593 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:10.962568    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"]
Oct 30 03:04:11.036833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.036801    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmtg\" (UniqueName: \"kubernetes.io/projected/e9c27684-504b-498c-92f5-8ca35b30b849-kube-api-access-mwmtg\") pod \"payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8\" (UID: \"e9c27684-504b-498c-92f5-8ca35b30b849\") " pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:04:11.037208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.036840    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c27684-504b-498c-92f5-8ca35b30b849-plugins-config-volume\") pod \"payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8\" (UID: \"e9c27684-504b-498c-92f5-8ca35b30b849\") " pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:04:11.138607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.138526    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmtg\" (UniqueName: \"kubernetes.io/projected/e9c27684-504b-498c-92f5-8ca35b30b849-kube-api-access-mwmtg\") pod \"payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8\" (UID: \"e9c27684-504b-498c-92f5-8ca35b30b849\") " pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:04:11.138607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.138567    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c27684-504b-498c-92f5-8ca35b30b849-plugins-config-volume\") pod \"payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8\" (UID: \"e9c27684-504b-498c-92f5-8ca35b30b849\") " pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:04:11.139323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.139304    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c27684-504b-498c-92f5-8ca35b30b849-plugins-config-volume\") pod \"payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8\" (UID: \"e9c27684-504b-498c-92f5-8ca35b30b849\") " pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:04:11.157528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.157497    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmtg\" (UniqueName: \"kubernetes.io/projected/e9c27684-504b-498c-92f5-8ca35b30b849-kube-api-access-mwmtg\") pod \"payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8\" (UID: \"e9c27684-504b-498c-92f5-8ca35b30b849\") " pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:04:11.223678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.223644    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:04:11.283561 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.283525    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:04:11.537404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.537378    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"]
Oct 30 03:04:11.539970 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:04:11.539940    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ca6e05_6e01_44e5_913a_c285f53af294.slice/crio-d7fdd43e58b5a476eff547ee95aca0f40c53a68610fa777fdfea84a505189544 WatchSource:0}: Error finding container d7fdd43e58b5a476eff547ee95aca0f40c53a68610fa777fdfea84a505189544: Status 404 returned error can't find the container with id d7fdd43e58b5a476eff547ee95aca0f40c53a68610fa777fdfea84a505189544
Oct 30 03:04:11.613315 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.613289    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"]
Oct 30 03:04:11.614384 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:04:11.614358    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c27684_504b_498c_92f5_8ca35b30b849.slice/crio-ced327c2b110a46d0f0b4c0fcbe4a3017db3c486c5185af4421785e568493361 WatchSource:0}: Error finding container ced327c2b110a46d0f0b4c0fcbe4a3017db3c486c5185af4421785e568493361: Status 404 returned error can't find the container with id ced327c2b110a46d0f0b4c0fcbe4a3017db3c486c5185af4421785e568493361
Oct 30 03:04:11.998653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.998627    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:04:11.998810 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:11.998688    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:04:12.112473 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:12.112432    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd" event={"ID":"82ca6e05-6e01-44e5-913a-c285f53af294","Type":"ContainerStarted","Data":"d03d1d686f7a10ba665fb59b6bf4f5c858aa38802cb5343733492f0cc34d6230"}
Oct 30 03:04:12.112984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:12.112537    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd" event={"ID":"82ca6e05-6e01-44e5-913a-c285f53af294","Type":"ContainerStarted","Data":"d7fdd43e58b5a476eff547ee95aca0f40c53a68610fa777fdfea84a505189544"}
Oct 30 03:04:12.112984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:12.112682    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:04:12.115243 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:12.115210    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8" event={"ID":"e9c27684-504b-498c-92f5-8ca35b30b849","Type":"ContainerStarted","Data":"9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1"}
Oct 30 03:04:12.115365 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:12.115256    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8" event={"ID":"e9c27684-504b-498c-92f5-8ca35b30b849","Type":"ContainerStarted","Data":"ced327c2b110a46d0f0b4c0fcbe4a3017db3c486c5185af4421785e568493361"}
Oct 30 03:04:12.115911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:12.115871    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:04:12.137615 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:12.137589    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:12.150342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:12.150291    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd" podStartSLOduration=2.150279272 podStartE2EDuration="2.150279272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:12.148344363 +0000 UTC m=+1397.450522730" watchObservedRunningTime="2026-10-30 03:04:12.150279272 +0000 UTC m=+1397.452457636"
Oct 30 03:04:12.185539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:12.185496    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8" podStartSLOduration=2.185485093 podStartE2EDuration="2.185485093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:12.184942789 +0000 UTC m=+1397.487121154" watchObservedRunningTime="2026-10-30 03:04:12.185485093 +0000 UTC m=+1397.487663457"
Oct 30 03:04:13.769300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:13.769258    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:15.376547 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:15.376512    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:16.709801 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:16.709773    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:17.322944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:17.301474    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:17.459951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:17.450821    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:17.848608 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:17.848561    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:18.868488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:18.868455    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:20.593994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:20.593956    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:21.864467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:21.864431    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:21.997951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:21.997920    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:23.130089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:23.130062    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:04:23.337668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:23.337632    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:23.471070 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:23.470978    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:24.136242 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.136212    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:04:24.206887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.206823    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:24.338489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.338458    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"]
Oct 30 03:04:24.343689 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.343664    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:04:24.344448 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.344423    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:24.359052 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.359030    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d30e9f\""
Oct 30 03:04:24.362982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.362949    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d30e9f-dockercfg-j64ks\""
Oct 30 03:04:24.365283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.365261    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"]
Oct 30 03:04:24.469939 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.467045    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k5l4\" (UniqueName: \"kubernetes.io/projected/ccb0f190-1234-4db3-8cc8-e5a969636704-kube-api-access-5k5l4\") pod \"payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g\" (UID: \"ccb0f190-1234-4db3-8cc8-e5a969636704\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:04:24.469939 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.467295    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb0f190-1234-4db3-8cc8-e5a969636704-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g\" (UID: \"ccb0f190-1234-4db3-8cc8-e5a969636704\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:04:24.588886 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.579675    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb0f190-1234-4db3-8cc8-e5a969636704-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g\" (UID: \"ccb0f190-1234-4db3-8cc8-e5a969636704\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:04:24.588886 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.580096    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k5l4\" (UniqueName: \"kubernetes.io/projected/ccb0f190-1234-4db3-8cc8-e5a969636704-kube-api-access-5k5l4\") pod \"payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g\" (UID: \"ccb0f190-1234-4db3-8cc8-e5a969636704\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:04:24.634870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.634643    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb0f190-1234-4db3-8cc8-e5a969636704-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g\" (UID: \"ccb0f190-1234-4db3-8cc8-e5a969636704\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:04:24.762828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.757566    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k5l4\" (UniqueName: \"kubernetes.io/projected/ccb0f190-1234-4db3-8cc8-e5a969636704-kube-api-access-5k5l4\") pod \"payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g\" (UID: \"ccb0f190-1234-4db3-8cc8-e5a969636704\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:04:24.968257 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:24.968224    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:04:25.291428 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:25.291401    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"]
Oct 30 03:04:25.295483 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:04:25.294759    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb0f190_1234_4db3_8cc8_e5a969636704.slice/crio-761ae07c3ccfcf5f2e861bd4e6956381fe96693a654040bcaa9a6ab52d0b3b39 WatchSource:0}: Error finding container 761ae07c3ccfcf5f2e861bd4e6956381fe96693a654040bcaa9a6ab52d0b3b39: Status 404 returned error can't find the container with id 761ae07c3ccfcf5f2e861bd4e6956381fe96693a654040bcaa9a6ab52d0b3b39
Oct 30 03:04:25.558924 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:04:25.550061    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:04:26.263934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:26.263890    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g" event={"ID":"ccb0f190-1234-4db3-8cc8-e5a969636704","Type":"ContainerStarted","Data":"3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0"}
Oct 30 03:04:26.264145 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:26.263961    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g" event={"ID":"ccb0f190-1234-4db3-8cc8-e5a969636704","Type":"ContainerStarted","Data":"761ae07c3ccfcf5f2e861bd4e6956381fe96693a654040bcaa9a6ab52d0b3b39"}
Oct 30 03:04:26.264326 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:26.264301    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:04:26.302394 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:26.302338    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g" podStartSLOduration=2.302323118 podStartE2EDuration="2.302323118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:26.300835922 +0000 UTC m=+1411.603014291" watchObservedRunningTime="2026-10-30 03:04:26.302323118 +0000 UTC m=+1411.604501484"
Oct 30 03:04:26.887569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:26.887515    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:28.900088 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:28.900051    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:29.755260 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:29.755218    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:30.484091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:30.484060    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:30.872803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:30.872767    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:31.123976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.123888    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"]
Oct 30 03:04:31.130364 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.130333    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.149500 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.149476    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"]
Oct 30 03:04:31.153476 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.153455    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c1abc2-openshift-default-dockercfg-2s9hb\""
Oct 30 03:04:31.237547 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.237510    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-envoy\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.237743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.237558    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-credential-socket\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.237743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.237693    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-socket\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.237914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.237746    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-podinfo\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.238004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.237974    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add485b5-1cb9-4342-8fcc-838fe468aaa7-istiod-ca-cert\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.238059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.238038    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-data\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.238156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.238139    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-token\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.238222 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.238205    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-certs\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.238309 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.238289    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbn6\" (UniqueName: \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-kube-api-access-4cbn6\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.265340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.265309    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:31.339661 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.339623    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add485b5-1cb9-4342-8fcc-838fe468aaa7-istiod-ca-cert\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.339869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.339689    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-data\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.339869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.339752    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-token\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.339869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.339818    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-certs\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.340058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.339896    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbn6\" (UniqueName: \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-kube-api-access-4cbn6\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.340058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.339965    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-envoy\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.340058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.340011    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-credential-socket\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.340223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.340081    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-socket\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.340223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.340121    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-podinfo\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.340223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.340210    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-data\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.340392 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.340374    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-credential-socket\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.340505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.340480    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-socket\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.340618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.340600    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-certs\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.340715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.340696    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add485b5-1cb9-4342-8fcc-838fe468aaa7-istiod-ca-cert\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.342809 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.342780    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-envoy\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.343004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.342970    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-podinfo\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.358966 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.358944    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-token\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.359510 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.359492    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbn6\" (UniqueName: \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-kube-api-access-4cbn6\") pod \"e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") " pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.403277 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.403190    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:31.450625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.450345    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:31.821287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.821260    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"]
Oct 30 03:04:31.823686 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:04:31.823660    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd485b5_1cb9_4342_8fcc_838fe468aaa7.slice/crio-48c00bb8718a5381b21b8b2b57c4f9a2d35a5c830fb583d49160ba34c949af20 WatchSource:0}: Error finding container 48c00bb8718a5381b21b8b2b57c4f9a2d35a5c830fb583d49160ba34c949af20: Status 404 returned error can't find the container with id 48c00bb8718a5381b21b8b2b57c4f9a2d35a5c830fb583d49160ba34c949af20
Oct 30 03:04:31.827440 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.827406    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:04:31.827536 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.827470    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:04:31.827536 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:31.827500    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:04:32.319251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:32.319204    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" event={"ID":"add485b5-1cb9-4342-8fcc-838fe468aaa7","Type":"ContainerStarted","Data":"c782a9dd35f39524c4e859b9f8f16aebcf3688a00db523b490b3def4406faa69"}
Oct 30 03:04:32.319425 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:32.319280    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" event={"ID":"add485b5-1cb9-4342-8fcc-838fe468aaa7","Type":"ContainerStarted","Data":"48c00bb8718a5381b21b8b2b57c4f9a2d35a5c830fb583d49160ba34c949af20"}
Oct 30 03:04:32.371329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:32.371279    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" podStartSLOduration=1.371265652 podStartE2EDuration="1.371265652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:32.368211252 +0000 UTC m=+1417.670389617" watchObservedRunningTime="2026-10-30 03:04:32.371265652 +0000 UTC m=+1417.673444016"
Oct 30 03:04:32.451197 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:32.451164    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:32.456959 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:32.456933    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:32.762267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:32.762175    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:33.327523 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:33.327486    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:33.328554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:33.328527    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:04:34.566204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:34.566166    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:34.702828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:34.702797    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:34.813343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:34.813306    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:35.071033 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:35.071001    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:35.276372 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:35.276333    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:36.807646 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:36.807606    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:36.946555 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:36.946514    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:37.291718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:37.291634    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:04:38.712003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:38.711964    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:40.440243 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:40.440210    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:42.062710 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:42.062672    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:43.636266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:43.636226    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:45.455029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:45.454999    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:45.581565 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:45.581512    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:47.204348 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:47.204311    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:47.372772 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:47.372738    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:49.110257 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:49.110225    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:51.420316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:51.420276    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:53.573872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:53.567505    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:55.154113 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:55.154066    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:55.471987 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:04:55.471906    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:04:56.984021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:56.983983    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:58.623482 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:04:58.623445    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:00.161292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:00.161250    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:00.281715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:00.281678    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:01.801164 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:01.801117    5061 patch_prober.go:29] interesting pod/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:05:01.801650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:01.801195    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:02.387533 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:02.387479    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:02.689283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:02.689187    5061 generic.go:356] "Generic (PLEG): container finished" podID="059a5e38-5767-417c-93d4-54b7af112f2f" containerID="7cae930294489af94369add82e07fa846162e2e3089ec58e3a08163a064352ee" exitCode=0
Oct 30 03:05:02.689481 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:02.689270    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" event={"ID":"059a5e38-5767-417c-93d4-54b7af112f2f","Type":"ContainerDied","Data":"7cae930294489af94369add82e07fa846162e2e3089ec58e3a08163a064352ee"}
Oct 30 03:05:02.689766 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:02.689747    5061 scope.go:122] "RemoveContainer" containerID="7cae930294489af94369add82e07fa846162e2e3089ec58e3a08163a064352ee"
Oct 30 03:05:02.691495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:02.691466    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:05:02.691583 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:02.691536    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:05:02.691583 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:02.691567    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:05:03.139649 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:03.139613    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:03.659060 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:03.654180    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:03.816868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:03.810820    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" event={"ID":"059a5e38-5767-417c-93d4-54b7af112f2f","Type":"ContainerStarted","Data":"e087378047f87e4dd1c5cee49d9fd58f441bdc3a40958a11771b5a3e4494cd46"}
Oct 30 03:05:03.855871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:03.855002    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:04.588172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:04.588142    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:05:04.588622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:04.588295    5061 patch_prober.go:29] interesting pod/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:05:04.588622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:04.588357    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 03:05:04.645596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:04.645419    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:05.589832 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:05.589290    5061 patch_prober.go:29] interesting pod/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:05:05.589832 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:05.589349    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 03:05:06.446869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:06.446712    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:06.577876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:06.577659    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:06.591616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:06.589544    5061 patch_prober.go:29] interesting pod/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:05:06.591616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:06.589599    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 03:05:06.703565 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:06.703450    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:07.587801 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:07.587764    5061 patch_prober.go:29] interesting pod/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:05:07.587994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:07.587815    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 03:05:08.010106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:08.010027    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:08.145175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:08.145007    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:08.588321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:08.588285    5061 patch_prober.go:29] interesting pod/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 03:05:08.588519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:08.588334    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 03:05:08.912612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:08.912524    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:09.172707 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:09.172603    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:09.592300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:09.592271    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:05:09.592654 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:09.592628    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:05:09.593266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:09.593246    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:05:09.778704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:09.778659    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:09.906004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:09.905923    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:10.057100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:10.057064    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:10.271924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:10.257771    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:10.716186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:10.716150    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:11.024908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:11.024812    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:12.755252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:12.755206    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:13.337987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:13.337956    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"]
Oct 30 03:05:13.366035 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:13.366007    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"]
Oct 30 03:05:13.501483 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:13.501453    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:13.627479 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:13.627385    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:14.941479 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:14.941440    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:15.035999 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:15.035962    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"]
Oct 30 03:05:15.057171 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:15.057052    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999a8ab0-7w4ws"]
Oct 30 03:05:15.062229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:15.062182    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:15.176004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:15.175967    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:16.721563 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:16.721528    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:16.850386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:16.850346    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1107a11-8162-46a6-ba4b-4387e19705d5" path="/var/lib/kubelet/pods/c1107a11-8162-46a6-ba4b-4387e19705d5/volumes"
Oct 30 03:05:17.308120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:17.308087    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:18.339183 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:18.339142    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp" podUID="e6dcb7ed-b1cc-49d4-a033-eafd0dd90339" containerName="payload-processing" containerID="cri-o://70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8" gracePeriod=25
Oct 30 03:05:18.366836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:18.366798    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc" podUID="311182e9-9db5-4c05-969b-5ea5a94f29f0" containerName="payload-pre-processing" containerID="cri-o://26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954" gracePeriod=25
Oct 30 03:05:18.735450 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:18.735377    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:19.098976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:19.098940    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:20.525174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:20.525127    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:20.668868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:20.661685    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:20.700872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:20.692295    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc" podUID="311182e9-9db5-4c05-969b-5ea5a94f29f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:05:20.808872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:20.805686    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:21.671792 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:21.671747    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp" podUID="e6dcb7ed-b1cc-49d4-a033-eafd0dd90339" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:05:21.885894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:21.885822    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:22.009339 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:22.009265    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:22.159121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:22.159079    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:22.669667 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:22.669610    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:24.423260 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:24.423223    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:25.447892 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:05:25.447837    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:05:25.876123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:25.876091    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:27.809796 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:27.809759    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:29.773441 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:29.773400    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:30.655601 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:30.655559    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc" podUID="311182e9-9db5-4c05-969b-5ea5a94f29f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:05:30.701495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:30.701469    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:31.419301 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:31.419269    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:31.671568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:31.671491    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp" podUID="e6dcb7ed-b1cc-49d4-a033-eafd0dd90339" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:05:31.678022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:31.677997    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:32.230246 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:32.230198    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:32.602934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:32.602895    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:33.671683 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:33.671646    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:34.245230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:34.245192    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:35.289074 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:35.288997    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:36.083047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:36.083018    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:36.200933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:36.200895    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:37.154509 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:37.154473    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:37.803236 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:37.803191    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:37.946615 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:37.946583    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:38.640940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.640912    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:05:38.726170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.726140    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:05:38.777784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.777747    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-kube-api-access-t4p4w\" (UniqueName: \"kubernetes.io/projected/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-kube-api-access-t4p4w\") pod \"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339\" (UID: \"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339\") "
Oct 30 03:05:38.778169 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.778146    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-plugins-config-volume\") pod \"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339\" (UID: \"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339\") "
Oct 30 03:05:38.781753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.781722    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-plugins-config-volume" pod "e6dcb7ed-b1cc-49d4-a033-eafd0dd90339" (UID: "e6dcb7ed-b1cc-49d4-a033-eafd0dd90339"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:38.784190 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.784162    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-kube-api-access-t4p4w" pod "e6dcb7ed-b1cc-49d4-a033-eafd0dd90339" (UID: "e6dcb7ed-b1cc-49d4-a033-eafd0dd90339"). InnerVolumeSpecName "kube-api-access-t4p4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:38.880528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.880483    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311182e9-9db5-4c05-969b-5ea5a94f29f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311182e9-9db5-4c05-969b-5ea5a94f29f0-plugins-config-volume\") pod \"311182e9-9db5-4c05-969b-5ea5a94f29f0\" (UID: \"311182e9-9db5-4c05-969b-5ea5a94f29f0\") "
Oct 30 03:05:38.880692 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.880676    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311182e9-9db5-4c05-969b-5ea5a94f29f0-kube-api-access-c6d5n\" (UniqueName: \"kubernetes.io/projected/311182e9-9db5-4c05-969b-5ea5a94f29f0-kube-api-access-c6d5n\") pod \"311182e9-9db5-4c05-969b-5ea5a94f29f0\" (UID: \"311182e9-9db5-4c05-969b-5ea5a94f29f0\") "
Oct 30 03:05:38.881010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.880980    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311182e9-9db5-4c05-969b-5ea5a94f29f0-plugins-config-volume" pod "311182e9-9db5-4c05-969b-5ea5a94f29f0" (UID: "311182e9-9db5-4c05-969b-5ea5a94f29f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:38.881549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.881509    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4p4w\" (UniqueName: \"kubernetes.io/projected/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-kube-api-access-t4p4w\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:05:38.881627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.881556    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311182e9-9db5-4c05-969b-5ea5a94f29f0-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:05:38.881627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.881574    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:05:38.883504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.883482    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311182e9-9db5-4c05-969b-5ea5a94f29f0-kube-api-access-c6d5n" pod "311182e9-9db5-4c05-969b-5ea5a94f29f0" (UID: "311182e9-9db5-4c05-969b-5ea5a94f29f0"). InnerVolumeSpecName "kube-api-access-c6d5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:38.983151 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:38.983115    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6d5n\" (UniqueName: \"kubernetes.io/projected/311182e9-9db5-4c05-969b-5ea5a94f29f0-kube-api-access-c6d5n\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:05:39.123300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.123218    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:39.438870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.435263    5061 generic.go:356] "Generic (PLEG): container finished" podID="e6dcb7ed-b1cc-49d4-a033-eafd0dd90339" containerID="70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8" exitCode=0
Oct 30 03:05:39.438870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.435389    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp" event={"ID":"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339","Type":"ContainerDied","Data":"70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8"}
Oct 30 03:05:39.438870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.435439    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp" event={"ID":"e6dcb7ed-b1cc-49d4-a033-eafd0dd90339","Type":"ContainerDied","Data":"7caec70187a3b04f32d0fa7765ee8eabaebfbc7d1913d41877c33a1f088bcc00"}
Oct 30 03:05:39.438870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.435468    5061 scope.go:122] "RemoveContainer" containerID="70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8"
Oct 30 03:05:39.438870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.435621    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"
Oct 30 03:05:39.450869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.445340    5061 generic.go:356] "Generic (PLEG): container finished" podID="311182e9-9db5-4c05-969b-5ea5a94f29f0" containerID="26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954" exitCode=0
Oct 30 03:05:39.450869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.445635    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc" event={"ID":"311182e9-9db5-4c05-969b-5ea5a94f29f0","Type":"ContainerDied","Data":"26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954"}
Oct 30 03:05:39.450869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.449960    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc" event={"ID":"311182e9-9db5-4c05-969b-5ea5a94f29f0","Type":"ContainerDied","Data":"c2bcbb39a280bded20b7c0c94510adf87116ded3fc56c071b7f5d43952f87a94"}
Oct 30 03:05:39.450869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.450054    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"
Oct 30 03:05:39.530911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.530881    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"]
Oct 30 03:05:39.540109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.540002    5061 scope.go:122] "RemoveContainer" containerID="70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8"
Oct 30 03:05:39.540712 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:05:39.540567    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8\": container with ID starting with 70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8 not found: ID does not exist" containerID="70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8"
Oct 30 03:05:39.540712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.540602    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8"} err="failed to get container status \"70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8\": rpc error: code = NotFound desc = could not find container \"70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8\": container with ID starting with 70637714cf7bb5ef4b87ce2ada3a006f46be74805bb6d65322649bf5ba2917a8 not found: ID does not exist"
Oct 30 03:05:39.540712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.540629    5061 scope.go:122] "RemoveContainer" containerID="26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954"
Oct 30 03:05:39.544226 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.544205    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-be9f2a59-75b9f5b854-5zlvp"]
Oct 30 03:05:39.553314 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.553226    5061 scope.go:122] "RemoveContainer" containerID="26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954"
Oct 30 03:05:39.553592 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:05:39.553566    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954\": container with ID starting with 26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954 not found: ID does not exist" containerID="26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954"
Oct 30 03:05:39.553689 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.553597    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954"} err="failed to get container status \"26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954\": rpc error: code = NotFound desc = could not find container \"26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954\": container with ID starting with 26c0356e0a028bec083a4db474b15319c85313f3344f258aa0c068fe431d4954 not found: ID does not exist"
Oct 30 03:05:39.560165 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.560143    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"]
Oct 30 03:05:39.573785 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:39.573760    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-be9f2a59-75665fcbff-825bc"]
Oct 30 03:05:40.659217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:40.659128    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:40.864743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:40.864698    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311182e9-9db5-4c05-969b-5ea5a94f29f0" path="/var/lib/kubelet/pods/311182e9-9db5-4c05-969b-5ea5a94f29f0/volumes"
Oct 30 03:05:40.865390 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:40.865367    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6dcb7ed-b1cc-49d4-a033-eafd0dd90339" path="/var/lib/kubelet/pods/e6dcb7ed-b1cc-49d4-a033-eafd0dd90339/volumes"
Oct 30 03:05:41.370467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:41.370423    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:42.844871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:42.844216    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.009174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.009129    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.153955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.151005    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.292871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.269707    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:05:43.411962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.411476    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.467319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.467260    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.467943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.467904    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 03:05:43.468423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.468398    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 03:05:43.859881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.821062    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Oct 30 03:05:43.904757 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.867659    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.905163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.867755    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:43.906479 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.906448    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 03:05:43.961319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:43.961273    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.012410 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:44.012369    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:44.017520 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:44.017494    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 03:05:44.109637 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:44.109594    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 03:05:44.204667 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:44.203997    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 03:05:44.433267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:44.433232    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:47.338483 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:47.338450    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:47.625292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:47.625202    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:48.750331 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:48.750291    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:49.110460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:49.104508    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:49.227394 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:49.227358    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:49.369045 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:49.368950    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:05:49.369045 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:49.369015    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:05:49.426527 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:49.426473    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:49.719913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:49.719776    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:50.178411 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:50.178363    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:50.986059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:50.986029    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:51.189660 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:51.189570    5061 generic.go:356] "Generic (PLEG): container finished" podID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerID="c782a9dd35f39524c4e859b9f8f16aebcf3688a00db523b490b3def4406faa69" exitCode=0
Oct 30 03:05:51.190073 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:51.189670    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" event={"ID":"add485b5-1cb9-4342-8fcc-838fe468aaa7","Type":"ContainerDied","Data":"c782a9dd35f39524c4e859b9f8f16aebcf3688a00db523b490b3def4406faa69"}
Oct 30 03:05:51.190259 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:51.190214    5061 scope.go:122] "RemoveContainer" containerID="c782a9dd35f39524c4e859b9f8f16aebcf3688a00db523b490b3def4406faa69"
Oct 30 03:05:51.192362 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:51.192329    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:05:51.192472 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:51.192418    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:05:51.192472 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:51.192466    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:05:52.200949 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:52.200916    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" event={"ID":"add485b5-1cb9-4342-8fcc-838fe468aaa7","Type":"ContainerStarted","Data":"5d36d7298164fc92ef58b4639cf7ebd0322293b181e13901a04e7497dcb73b9d"}
Oct 30 03:05:52.451393 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:52.451313    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:05:52.451543 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:52.451506    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 03:05:52.451596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:52.451573    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 03:05:52.790687 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:52.790605    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:53.460870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:53.458303    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 03:05:53.460870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:53.458370    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 03:05:53.806242 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:53.806184    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:53.958128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:53.958092    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:54.451558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:54.451519    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 03:05:54.451742 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:54.451595    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 03:05:54.848279 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:54.848217    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:55.135999 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:55.135911    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:05:55.145907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:55.138905    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:05:55.152313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:55.152294    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:05:55.152457 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:55.152296    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:05:55.452350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:55.452267    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 03:05:55.452350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:55.452328    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 03:05:55.475474 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:05:55.475441    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:05:55.651064 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:55.650981    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:55.897623 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:55.897587    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:56.459815 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:56.459777    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 03:05:56.460201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:56.460165    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 03:05:57.058789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:57.058749    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:57.452568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:57.452470    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 03:05:57.452568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:57.452535    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 03:05:58.455620 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:58.455590    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:05:58.456117 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:58.456099    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:05:58.456688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:58.456672    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:05:59.002053 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:05:59.002019    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:00.895907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:00.895874    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:02.020015 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:02.019982    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:02.796437 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:02.796407    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:04.592958 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:04.592924    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:06.314815 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:06.314780    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:08.176150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:08.176112    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:09.779669 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:09.779600    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:11.392308 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:11.392266    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:13.099961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:13.099927    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:13.904345 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:13.904314    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"]
Oct 30 03:06:13.935803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:13.935770    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"]
Oct 30 03:06:14.074824 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:14.074791    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:14.198368 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:14.198255    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:14.740550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:14.740516    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:16.428467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:16.428430    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:18.405505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:18.405465    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:18.538871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:18.537817    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:18.657630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:18.657522    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:18.904918 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:18.904840    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj" podUID="59d8f59d-ff26-4c89-8c35-896b6bda6b75" containerName="payload-processing" containerID="cri-o://bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba" gracePeriod=25
Oct 30 03:06:18.936597 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:18.936547    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj" podUID="d6c421f7-50d6-4401-941d-542df467aa4f" containerName="payload-pre-processing" containerID="cri-o://a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5" gracePeriod=25
Oct 30 03:06:19.287086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.287059    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:06:19.356023 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.355989    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59d8f59d-ff26-4c89-8c35-896b6bda6b75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d8f59d-ff26-4c89-8c35-896b6bda6b75-plugins-config-volume\") pod \"59d8f59d-ff26-4c89-8c35-896b6bda6b75\" (UID: \"59d8f59d-ff26-4c89-8c35-896b6bda6b75\") "
Oct 30 03:06:19.356165 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.356077    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59d8f59d-ff26-4c89-8c35-896b6bda6b75-kube-api-access-dspc5\" (UniqueName: \"kubernetes.io/projected/59d8f59d-ff26-4c89-8c35-896b6bda6b75-kube-api-access-dspc5\") pod \"59d8f59d-ff26-4c89-8c35-896b6bda6b75\" (UID: \"59d8f59d-ff26-4c89-8c35-896b6bda6b75\") "
Oct 30 03:06:19.356418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.356385    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d8f59d-ff26-4c89-8c35-896b6bda6b75-plugins-config-volume" pod "59d8f59d-ff26-4c89-8c35-896b6bda6b75" (UID: "59d8f59d-ff26-4c89-8c35-896b6bda6b75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:19.357053 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.357029    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d8f59d-ff26-4c89-8c35-896b6bda6b75-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:06:19.358607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.358581    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d8f59d-ff26-4c89-8c35-896b6bda6b75-kube-api-access-dspc5" pod "59d8f59d-ff26-4c89-8c35-896b6bda6b75" (UID: "59d8f59d-ff26-4c89-8c35-896b6bda6b75"). InnerVolumeSpecName "kube-api-access-dspc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:19.360413 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.360393    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:06:19.457554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.457515    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c421f7-50d6-4401-941d-542df467aa4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c421f7-50d6-4401-941d-542df467aa4f-plugins-config-volume\") pod \"d6c421f7-50d6-4401-941d-542df467aa4f\" (UID: \"d6c421f7-50d6-4401-941d-542df467aa4f\") "
Oct 30 03:06:19.458040 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.457689    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c421f7-50d6-4401-941d-542df467aa4f-kube-api-access-pg4gb\" (UniqueName: \"kubernetes.io/projected/d6c421f7-50d6-4401-941d-542df467aa4f-kube-api-access-pg4gb\") pod \"d6c421f7-50d6-4401-941d-542df467aa4f\" (UID: \"d6c421f7-50d6-4401-941d-542df467aa4f\") "
Oct 30 03:06:19.458040 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.458014    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c421f7-50d6-4401-941d-542df467aa4f-plugins-config-volume" pod "d6c421f7-50d6-4401-941d-542df467aa4f" (UID: "d6c421f7-50d6-4401-941d-542df467aa4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:19.458566 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.458542    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dspc5\" (UniqueName: \"kubernetes.io/projected/59d8f59d-ff26-4c89-8c35-896b6bda6b75-kube-api-access-dspc5\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:06:19.458631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.458572    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c421f7-50d6-4401-941d-542df467aa4f-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:06:19.460108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.460088    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c421f7-50d6-4401-941d-542df467aa4f-kube-api-access-pg4gb" pod "d6c421f7-50d6-4401-941d-542df467aa4f" (UID: "d6c421f7-50d6-4401-941d-542df467aa4f"). InnerVolumeSpecName "kube-api-access-pg4gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:19.561540 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.561383    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg4gb\" (UniqueName: \"kubernetes.io/projected/d6c421f7-50d6-4401-941d-542df467aa4f-kube-api-access-pg4gb\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:06:19.766908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.766870    5061 generic.go:356] "Generic (PLEG): container finished" podID="d6c421f7-50d6-4401-941d-542df467aa4f" containerID="a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5" exitCode=0
Oct 30 03:06:19.767375 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.767349    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj" event={"ID":"d6c421f7-50d6-4401-941d-542df467aa4f","Type":"ContainerDied","Data":"a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5"}
Oct 30 03:06:19.767587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.767571    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj" event={"ID":"d6c421f7-50d6-4401-941d-542df467aa4f","Type":"ContainerDied","Data":"66e6dbcd95f009ad50301a97d82de0e1884701905ca0cde72ad3da6029ea9a9f"}
Oct 30 03:06:19.767715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.767702    5061 scope.go:122] "RemoveContainer" containerID="a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5"
Oct 30 03:06:19.767971 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.767953    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"
Oct 30 03:06:19.773340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.773317    5061 generic.go:356] "Generic (PLEG): container finished" podID="59d8f59d-ff26-4c89-8c35-896b6bda6b75" containerID="bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba" exitCode=0
Oct 30 03:06:19.773449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.773383    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"
Oct 30 03:06:19.773449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.773408    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj" event={"ID":"59d8f59d-ff26-4c89-8c35-896b6bda6b75","Type":"ContainerDied","Data":"bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba"}
Oct 30 03:06:19.773547 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.773451    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj" event={"ID":"59d8f59d-ff26-4c89-8c35-896b6bda6b75","Type":"ContainerDied","Data":"ccc889273b2717ab0e2ee985b33da68bd434900ee0bffa44165875199f691308"}
Oct 30 03:06:19.784014 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.783992    5061 scope.go:122] "RemoveContainer" containerID="a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5"
Oct 30 03:06:19.784279 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:06:19.784258    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5\": container with ID starting with a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5 not found: ID does not exist" containerID="a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5"
Oct 30 03:06:19.784361 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.784286    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5"} err="failed to get container status \"a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5\": rpc error: code = NotFound desc = could not find container \"a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5\": container with ID starting with a16d20dd38587510260b99446cfd5e9f100624f731ca5a633b666c633e6599f5 not found: ID does not exist"
Oct 30 03:06:19.784361 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.784302    5061 scope.go:122] "RemoveContainer" containerID="bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba"
Oct 30 03:06:19.794455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.794439    5061 scope.go:122] "RemoveContainer" containerID="bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba"
Oct 30 03:06:19.794738 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:06:19.794706    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba\": container with ID starting with bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba not found: ID does not exist" containerID="bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba"
Oct 30 03:06:19.794790 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.794746    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba"} err="failed to get container status \"bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba\": rpc error: code = NotFound desc = could not find container \"bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba\": container with ID starting with bcc8d33b5fc71cf3b227963624af4366c34780a471bff43c1d0cb5001bc8a8ba not found: ID does not exist"
Oct 30 03:06:19.823328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.823282    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"]
Oct 30 03:06:19.833445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.833424    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d7fde0-857bc97f7f-4hssj"]
Oct 30 03:06:19.862079 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.862047    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"]
Oct 30 03:06:19.868316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:19.868271    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fde0-7f45c4bf74-4zggj"]
Oct 30 03:06:20.196563 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:20.196483    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:20.850020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:20.849991    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59d8f59d-ff26-4c89-8c35-896b6bda6b75" path="/var/lib/kubelet/pods/59d8f59d-ff26-4c89-8c35-896b6bda6b75/volumes"
Oct 30 03:06:20.850482 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:20.850466    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c421f7-50d6-4401-941d-542df467aa4f" path="/var/lib/kubelet/pods/d6c421f7-50d6-4401-941d-542df467aa4f/volumes"
Oct 30 03:06:21.865353 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:21.865322    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:23.573452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:23.573412    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:25.260869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:25.260038    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:25.540300 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:06:25.540212    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:06:26.928536 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:26.928499    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:27.646486 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:27.645707    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:27.912525 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:27.912429    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:28.207067 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:28.206292    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:31.131821 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:31.131789    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:32.310403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:32.310362    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:33.913766 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:33.913737    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:36.067026 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:36.066981    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:36.200372 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:36.200337    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:36.328674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:36.328592    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:38.065783 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:38.065705    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:38.512412 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:38.512281    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:38.665162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:38.665127    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:39.878608 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:39.878576    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:40.052129 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:40.052100    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:40.736292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:40.736252    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:41.371674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:41.370169    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:41.540066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:41.527489    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:42.027557 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:42.027524    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:44.183670 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:44.183607    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:45.255928 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:45.255894    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:45.377154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:45.377112    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:46.230910 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:46.230870    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:46.921755 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:46.921719    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:47.307505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:47.307400    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:47.631231 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:47.631152    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:48.524560 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:48.524525    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:50.303195 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:50.303163    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:51.655620 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:51.655585    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"]
Oct 30 03:06:51.810494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:51.810454    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:51.935570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:51.935487    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:53.579685 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:53.579652    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:54.438077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:54.438043    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:54.560584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:54.560549    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:55.051337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:55.051307    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:55.452129 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:06:55.452049    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:06:56.422565 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:56.422527    5061 scope.go:122] "RemoveContainer" containerID="f71f41bcbb9b29bc824cc6c2a0dea7eed05d77f87ad9b0ac890525acb8616fcd"
Oct 30 03:06:56.533984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:56.533953    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:56.656924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:56.656880    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4" podUID="1848202f-6bd1-4c94-9e2b-b8627325c908" containerName="payload-processing" containerID="cri-o://3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10" gracePeriod=25
Oct 30 03:06:57.065516 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.065080    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:06:57.087870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.084403    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1848202f-6bd1-4c94-9e2b-b8627325c908-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1848202f-6bd1-4c94-9e2b-b8627325c908-plugins-config-volume\") pod \"1848202f-6bd1-4c94-9e2b-b8627325c908\" (UID: \"1848202f-6bd1-4c94-9e2b-b8627325c908\") "
Oct 30 03:06:57.087870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.084457    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1848202f-6bd1-4c94-9e2b-b8627325c908-kube-api-access-xw9r4\" (UniqueName: \"kubernetes.io/projected/1848202f-6bd1-4c94-9e2b-b8627325c908-kube-api-access-xw9r4\") pod \"1848202f-6bd1-4c94-9e2b-b8627325c908\" (UID: \"1848202f-6bd1-4c94-9e2b-b8627325c908\") "
Oct 30 03:06:57.087870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.087142    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1848202f-6bd1-4c94-9e2b-b8627325c908-plugins-config-volume" pod "1848202f-6bd1-4c94-9e2b-b8627325c908" (UID: "1848202f-6bd1-4c94-9e2b-b8627325c908"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:06:57.127467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.127427    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1848202f-6bd1-4c94-9e2b-b8627325c908-kube-api-access-xw9r4" pod "1848202f-6bd1-4c94-9e2b-b8627325c908" (UID: "1848202f-6bd1-4c94-9e2b-b8627325c908"). InnerVolumeSpecName "kube-api-access-xw9r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:06:57.191777 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.191264    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1848202f-6bd1-4c94-9e2b-b8627325c908-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:06:57.191777 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.191299    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw9r4\" (UniqueName: \"kubernetes.io/projected/1848202f-6bd1-4c94-9e2b-b8627325c908-kube-api-access-xw9r4\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:06:57.318929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.317363    5061 generic.go:356] "Generic (PLEG): container finished" podID="1848202f-6bd1-4c94-9e2b-b8627325c908" containerID="3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10" exitCode=0
Oct 30 03:06:57.318929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.317432    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"
Oct 30 03:06:57.318929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.317469    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4" event={"ID":"1848202f-6bd1-4c94-9e2b-b8627325c908","Type":"ContainerDied","Data":"3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10"}
Oct 30 03:06:57.318929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.317513    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4" event={"ID":"1848202f-6bd1-4c94-9e2b-b8627325c908","Type":"ContainerDied","Data":"3702b43182b2e6faed20387919dd26635768bc2f7bcbc9bff8e26c57f0d1f58f"}
Oct 30 03:06:57.318929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.317541    5061 scope.go:122] "RemoveContainer" containerID="3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10"
Oct 30 03:06:57.341415 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.341347    5061 scope.go:122] "RemoveContainer" containerID="3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10"
Oct 30 03:06:57.341831 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:06:57.341805    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10\": container with ID starting with 3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10 not found: ID does not exist" containerID="3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10"
Oct 30 03:06:57.341949 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.341843    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10"} err="failed to get container status \"3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10\": rpc error: code = NotFound desc = could not find container \"3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10\": container with ID starting with 3b005b35165721f7a6dda0d0448ec72f2fd007afd1b1f133829b28e65139cb10 not found: ID does not exist"
Oct 30 03:06:57.371542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.371503    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"]
Oct 30 03:06:57.378571 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:57.378540    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ffb1886-64788bf9bd-5zrf4"]
Oct 30 03:06:58.280825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:58.280785    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:58.968870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:58.966777    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1848202f-6bd1-4c94-9e2b-b8627325c908" path="/var/lib/kubelet/pods/1848202f-6bd1-4c94-9e2b-b8627325c908/volumes"
Oct 30 03:06:59.908793 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:06:59.908752    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:01.739608 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:01.739572    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:03.624411 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:03.624381    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:05.746379 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:05.746301    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:07.497503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:07.497470    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:08.264015 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:08.263981    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:09.498571 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:09.498539    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:10.738364 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:10.738323    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:11.437475 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:11.437439    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:13.609691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:13.609622    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:13.807416 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:13.807330    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:15.321151 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:15.321109    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:16.163735 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:16.163696    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:17.775106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:17.775073    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:18.317162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:18.317124    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:19.625237 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:19.625204    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:19.800437 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:19.800404    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:20.053308 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:20.053276    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"]
Oct 30 03:07:20.118095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:20.118062    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"]
Oct 30 03:07:20.309615 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:20.309535    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:20.486149 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:20.486118    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:20.638109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:20.638033    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:21.483636 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:21.483600    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:21.640963 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:21.640929    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:21.800469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:21.800437    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:22.287787 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:22.287672    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:23.226547 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:23.226516    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:23.552531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:23.552498    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:25.053915 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.053856    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8" podUID="e9c27684-504b-498c-92f5-8ca35b30b849" containerName="payload-processing" containerID="cri-o://9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1" gracePeriod=25
Oct 30 03:07:25.119069 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.119032    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd" podUID="82ca6e05-6e01-44e5-913a-c285f53af294" containerName="payload-pre-processing" containerID="cri-o://d03d1d686f7a10ba665fb59b6bf4f5c858aa38802cb5343733492f0cc34d6230" gracePeriod=25
Oct 30 03:07:25.340488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.340453    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:25.748253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.747093    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:07:25.751749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.751715    5061 generic.go:356] "Generic (PLEG): container finished" podID="82ca6e05-6e01-44e5-913a-c285f53af294" containerID="d03d1d686f7a10ba665fb59b6bf4f5c858aa38802cb5343733492f0cc34d6230" exitCode=0
Oct 30 03:07:25.752050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.751974    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd" event={"ID":"82ca6e05-6e01-44e5-913a-c285f53af294","Type":"ContainerDied","Data":"d03d1d686f7a10ba665fb59b6bf4f5c858aa38802cb5343733492f0cc34d6230"}
Oct 30 03:07:25.754330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.754307    5061 generic.go:356] "Generic (PLEG): container finished" podID="e9c27684-504b-498c-92f5-8ca35b30b849" containerID="9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1" exitCode=0
Oct 30 03:07:25.754531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.754511    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"
Oct 30 03:07:25.754679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.754659    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8" event={"ID":"e9c27684-504b-498c-92f5-8ca35b30b849","Type":"ContainerDied","Data":"9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1"}
Oct 30 03:07:25.754809 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.754792    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8" event={"ID":"e9c27684-504b-498c-92f5-8ca35b30b849","Type":"ContainerDied","Data":"ced327c2b110a46d0f0b4c0fcbe4a3017db3c486c5185af4421785e568493361"}
Oct 30 03:07:25.754960 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.754946    5061 scope.go:122] "RemoveContainer" containerID="9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1"
Oct 30 03:07:25.761110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.761089    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:07:25.766929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.766907    5061 scope.go:122] "RemoveContainer" containerID="9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1"
Oct 30 03:07:25.767211 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:07:25.767188    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1\": container with ID starting with 9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1 not found: ID does not exist" containerID="9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1"
Oct 30 03:07:25.767289 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.767216    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1"} err="failed to get container status \"9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1\": rpc error: code = NotFound desc = could not find container \"9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1\": container with ID starting with 9a8ad3cb72cdd1e2a8c960d65bbd7f9c18cb132875efcd2484d5c803aa06d0b1 not found: ID does not exist"
Oct 30 03:07:25.777971 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:07:25.777933    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:07:25.852983 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.852907    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ca6e05-6e01-44e5-913a-c285f53af294-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ca6e05-6e01-44e5-913a-c285f53af294-plugins-config-volume\") pod \"82ca6e05-6e01-44e5-913a-c285f53af294\" (UID: \"82ca6e05-6e01-44e5-913a-c285f53af294\") "
Oct 30 03:07:25.853125 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.852989    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c27684-504b-498c-92f5-8ca35b30b849-kube-api-access-mwmtg\" (UniqueName: \"kubernetes.io/projected/e9c27684-504b-498c-92f5-8ca35b30b849-kube-api-access-mwmtg\") pod \"e9c27684-504b-498c-92f5-8ca35b30b849\" (UID: \"e9c27684-504b-498c-92f5-8ca35b30b849\") "
Oct 30 03:07:25.853200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.853186    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c27684-504b-498c-92f5-8ca35b30b849-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c27684-504b-498c-92f5-8ca35b30b849-plugins-config-volume\") pod \"e9c27684-504b-498c-92f5-8ca35b30b849\" (UID: \"e9c27684-504b-498c-92f5-8ca35b30b849\") "
Oct 30 03:07:25.853238 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.853201    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ca6e05-6e01-44e5-913a-c285f53af294-plugins-config-volume" pod "82ca6e05-6e01-44e5-913a-c285f53af294" (UID: "82ca6e05-6e01-44e5-913a-c285f53af294"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:25.853286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.853235    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ca6e05-6e01-44e5-913a-c285f53af294-kube-api-access-gnbtr\" (UniqueName: \"kubernetes.io/projected/82ca6e05-6e01-44e5-913a-c285f53af294-kube-api-access-gnbtr\") pod \"82ca6e05-6e01-44e5-913a-c285f53af294\" (UID: \"82ca6e05-6e01-44e5-913a-c285f53af294\") "
Oct 30 03:07:25.853611 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.853580    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c27684-504b-498c-92f5-8ca35b30b849-plugins-config-volume" pod "e9c27684-504b-498c-92f5-8ca35b30b849" (UID: "e9c27684-504b-498c-92f5-8ca35b30b849"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:25.853859 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.853827    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c27684-504b-498c-92f5-8ca35b30b849-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:25.853925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.853888    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ca6e05-6e01-44e5-913a-c285f53af294-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:25.855139 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.855119    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c27684-504b-498c-92f5-8ca35b30b849-kube-api-access-mwmtg" pod "e9c27684-504b-498c-92f5-8ca35b30b849" (UID: "e9c27684-504b-498c-92f5-8ca35b30b849"). InnerVolumeSpecName "kube-api-access-mwmtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:25.855358 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.855337    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ca6e05-6e01-44e5-913a-c285f53af294-kube-api-access-gnbtr" pod "82ca6e05-6e01-44e5-913a-c285f53af294" (UID: "82ca6e05-6e01-44e5-913a-c285f53af294"). InnerVolumeSpecName "kube-api-access-gnbtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:25.955276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.955246    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnbtr\" (UniqueName: \"kubernetes.io/projected/82ca6e05-6e01-44e5-913a-c285f53af294-kube-api-access-gnbtr\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:25.955276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:25.955274    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwmtg\" (UniqueName: \"kubernetes.io/projected/e9c27684-504b-498c-92f5-8ca35b30b849-kube-api-access-mwmtg\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:26.124579 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:26.124508    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"]
Oct 30 03:07:26.133619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:26.133587    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-44a8fd-7544b9d8b4-d79n8"]
Oct 30 03:07:26.171063 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:26.171028    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:26.774290 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:26.774253    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd" event={"ID":"82ca6e05-6e01-44e5-913a-c285f53af294","Type":"ContainerDied","Data":"d7fdd43e58b5a476eff547ee95aca0f40c53a68610fa777fdfea84a505189544"}
Oct 30 03:07:26.774290 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:26.774272    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"
Oct 30 03:07:26.774574 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:26.774347    5061 scope.go:122] "RemoveContainer" containerID="d03d1d686f7a10ba665fb59b6bf4f5c858aa38802cb5343733492f0cc34d6230"
Oct 30 03:07:26.828207 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:26.828178    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"]
Oct 30 03:07:26.839933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:26.839909    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-44a8fd-7dc6bc45bc-kd9hd"]
Oct 30 03:07:26.849635 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:26.849609    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ca6e05-6e01-44e5-913a-c285f53af294" path="/var/lib/kubelet/pods/82ca6e05-6e01-44e5-913a-c285f53af294/volumes"
Oct 30 03:07:26.850064 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:26.850050    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c27684-504b-498c-92f5-8ca35b30b849" path="/var/lib/kubelet/pods/e9c27684-504b-498c-92f5-8ca35b30b849/volumes"
Oct 30 03:07:27.146870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:27.146418    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:27.336249 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:27.336205    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:27.692619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:27.692585    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"]
Oct 30 03:07:27.692878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:27.692822    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy" containerID="cri-o://e087378047f87e4dd1c5cee49d9fd58f441bdc3a40958a11771b5a3e4494cd46" gracePeriod=30
Oct 30 03:07:27.784986 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:27.784951    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:28.840304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:28.840273    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:29.374660 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:29.374595    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:29.593057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:29.593006    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:31.378281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:31.378247    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:31.689837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:31.689734    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:31.947087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:31.946953    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:32.270397 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:32.270288    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:32.854434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:32.854400    5061 generic.go:356] "Generic (PLEG): container finished" podID="059a5e38-5767-417c-93d4-54b7af112f2f" containerID="e087378047f87e4dd1c5cee49d9fd58f441bdc3a40958a11771b5a3e4494cd46" exitCode=0
Oct 30 03:07:32.854882 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:32.854457    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" event={"ID":"059a5e38-5767-417c-93d4-54b7af112f2f","Type":"ContainerDied","Data":"e087378047f87e4dd1c5cee49d9fd58f441bdc3a40958a11771b5a3e4494cd46"}
Oct 30 03:07:32.854882 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:32.854510    5061 scope.go:122] "RemoveContainer" containerID="7cae930294489af94369add82e07fa846162e2e3089ec58e3a08163a064352ee"
Oct 30 03:07:32.995518 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:32.995494    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:07:33.105700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.105137    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-kube-api-access-qzxbr\" (UniqueName: \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-kube-api-access-qzxbr\") pod \"059a5e38-5767-417c-93d4-54b7af112f2f\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") "
Oct 30 03:07:33.105700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.105211    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-socket\") pod \"059a5e38-5767-417c-93d4-54b7af112f2f\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") "
Oct 30 03:07:33.105700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.105361    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-envoy\") pod \"059a5e38-5767-417c-93d4-54b7af112f2f\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") "
Oct 30 03:07:33.105700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.105391    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-credential-socket\") pod \"059a5e38-5767-417c-93d4-54b7af112f2f\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") "
Oct 30 03:07:33.105700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.105492    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-data\") pod \"059a5e38-5767-417c-93d4-54b7af112f2f\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") "
Oct 30 03:07:33.105700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.105533    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-istio-token\" (UniqueName: \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-istio-token\") pod \"059a5e38-5767-417c-93d4-54b7af112f2f\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") "
Oct 30 03:07:33.107171 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.107145    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/059a5e38-5767-417c-93d4-54b7af112f2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059a5e38-5767-417c-93d4-54b7af112f2f-istio-podinfo\") pod \"059a5e38-5767-417c-93d4-54b7af112f2f\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") "
Oct 30 03:07:33.111033 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.108010    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-data" pod "059a5e38-5767-417c-93d4-54b7af112f2f" (UID: "059a5e38-5767-417c-93d4-54b7af112f2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:33.111033 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.108200    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-socket" pod "059a5e38-5767-417c-93d4-54b7af112f2f" (UID: "059a5e38-5767-417c-93d4-54b7af112f2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:33.111033 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.110603    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-credential-socket" pod "059a5e38-5767-417c-93d4-54b7af112f2f" (UID: "059a5e38-5767-417c-93d4-54b7af112f2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:33.111510 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.111477    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-certs" pod "059a5e38-5767-417c-93d4-54b7af112f2f" (UID: "059a5e38-5767-417c-93d4-54b7af112f2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:33.111619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.110954    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-certs\") pod \"059a5e38-5767-417c-93d4-54b7af112f2f\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") "
Oct 30 03:07:33.111783 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.111770    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059a5e38-5767-417c-93d4-54b7af112f2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059a5e38-5767-417c-93d4-54b7af112f2f-istiod-ca-cert\") pod \"059a5e38-5767-417c-93d4-54b7af112f2f\" (UID: \"059a5e38-5767-417c-93d4-54b7af112f2f\") "
Oct 30 03:07:33.120192 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.120049    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059a5e38-5767-417c-93d4-54b7af112f2f-istiod-ca-cert" pod "059a5e38-5767-417c-93d4-54b7af112f2f" (UID: "059a5e38-5767-417c-93d4-54b7af112f2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:07:33.121277 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.121255    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059a5e38-5767-417c-93d4-54b7af112f2f-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:33.121414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.121402    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:33.121506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.121495    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:33.121587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.121577    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:33.121663 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.121654    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:33.170100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.160982    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-envoy" pod "059a5e38-5767-417c-93d4-54b7af112f2f" (UID: "059a5e38-5767-417c-93d4-54b7af112f2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:33.174920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.170404    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-kube-api-access-qzxbr" pod "059a5e38-5767-417c-93d4-54b7af112f2f" (UID: "059a5e38-5767-417c-93d4-54b7af112f2f"). InnerVolumeSpecName "kube-api-access-qzxbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:33.174920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.170549    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-istio-token" pod "059a5e38-5767-417c-93d4-54b7af112f2f" (UID: "059a5e38-5767-417c-93d4-54b7af112f2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:33.223932 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.223483    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzxbr\" (UniqueName: \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-kube-api-access-qzxbr\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:33.223932 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.223525    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059a5e38-5767-417c-93d4-54b7af112f2f-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:33.223932 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.223552    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059a5e38-5767-417c-93d4-54b7af112f2f-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:33.269929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.265920    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/059a5e38-5767-417c-93d4-54b7af112f2f-istio-podinfo" pod "059a5e38-5767-417c-93d4-54b7af112f2f" (UID: "059a5e38-5767-417c-93d4-54b7af112f2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:07:33.325770 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.325738    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059a5e38-5767-417c-93d4-54b7af112f2f-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:07:33.863700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.863665    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"
Oct 30 03:07:33.864142 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.863700    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d" event={"ID":"059a5e38-5767-417c-93d4-54b7af112f2f","Type":"ContainerDied","Data":"ad039b6ee1fd2f2518408ac3859d96dedb17f6b9aa8a03bcbaa672df8986bbdd"}
Oct 30 03:07:33.864142 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.863752    5061 scope.go:122] "RemoveContainer" containerID="e087378047f87e4dd1c5cee49d9fd58f441bdc3a40958a11771b5a3e4494cd46"
Oct 30 03:07:33.923123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.923093    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"]
Oct 30 03:07:33.933251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:33.933227    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7ffb1886-openshift-default-d7d7f6d89-lbd9d"]
Oct 30 03:07:34.256507 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:34.256477    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:34.450869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:34.447841    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:34.851013 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:34.850980    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" path="/var/lib/kubelet/pods/059a5e38-5767-417c-93d4-54b7af112f2f/volumes"
Oct 30 03:07:36.432069 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:36.432041    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:37.674840 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:37.674809    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:37.808622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:37.808591    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:38.189430 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:38.178659    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:39.880150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:39.880109    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:41.743004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:41.742967    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:43.552815 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:43.552775    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:43.753873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:43.753417    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:45.870124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:45.870080    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:47.897424 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:47.897369    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:49.143190 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:49.143155    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:52.231592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:52.231556    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:54.791639 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:54.791603    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:55.447825 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:07:55.447790    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:07:57.161532 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:57.161476    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:59.290697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:07:59.290651    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:01.537318 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:01.537284    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"]
Oct 30 03:08:01.537769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:01.537479    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy" containerID="cri-o://b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93" gracePeriod=30
Oct 30 03:08:01.815722 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:01.815685    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:03.991032 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:03.990985    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:05.989174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:05.989132    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:06.838511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.838443    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 03:08:06.942244 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942215    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-socket\") pod \"17c5f62d-44a9-4d43-aff1-6f0191511107\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") "
Oct 30 03:08:06.942367 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942320    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-kube-api-access-8ctcf\" (UniqueName: \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-kube-api-access-8ctcf\") pod \"17c5f62d-44a9-4d43-aff1-6f0191511107\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") "
Oct 30 03:08:06.942424 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942369    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-podinfo\") pod \"17c5f62d-44a9-4d43-aff1-6f0191511107\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") "
Oct 30 03:08:06.942473 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942448    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-certs\") pod \"17c5f62d-44a9-4d43-aff1-6f0191511107\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") "
Oct 30 03:08:06.942538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942484    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c5f62d-44a9-4d43-aff1-6f0191511107-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c5f62d-44a9-4d43-aff1-6f0191511107-istiod-ca-cert\") pod \"17c5f62d-44a9-4d43-aff1-6f0191511107\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") "
Oct 30 03:08:06.942538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942504    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-socket" pod "17c5f62d-44a9-4d43-aff1-6f0191511107" (UID: "17c5f62d-44a9-4d43-aff1-6f0191511107"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:06.942648 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942586    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-envoy\") pod \"17c5f62d-44a9-4d43-aff1-6f0191511107\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") "
Oct 30 03:08:06.942828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942701    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-token\" (UniqueName: \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-token\") pod \"17c5f62d-44a9-4d43-aff1-6f0191511107\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") "
Oct 30 03:08:06.942828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942751    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-data\") pod \"17c5f62d-44a9-4d43-aff1-6f0191511107\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") "
Oct 30 03:08:06.942828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942762    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-certs" pod "17c5f62d-44a9-4d43-aff1-6f0191511107" (UID: "17c5f62d-44a9-4d43-aff1-6f0191511107"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:06.942828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942794    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-credential-socket\") pod \"17c5f62d-44a9-4d43-aff1-6f0191511107\" (UID: \"17c5f62d-44a9-4d43-aff1-6f0191511107\") "
Oct 30 03:08:06.943114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.942888    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c5f62d-44a9-4d43-aff1-6f0191511107-istiod-ca-cert" pod "17c5f62d-44a9-4d43-aff1-6f0191511107" (UID: "17c5f62d-44a9-4d43-aff1-6f0191511107"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:06.943114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.943046    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-data" pod "17c5f62d-44a9-4d43-aff1-6f0191511107" (UID: "17c5f62d-44a9-4d43-aff1-6f0191511107"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:06.943114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.943088    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-credential-socket" pod "17c5f62d-44a9-4d43-aff1-6f0191511107" (UID: "17c5f62d-44a9-4d43-aff1-6f0191511107"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:06.944255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.944222    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:08:06.944255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.944255    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:08:06.944429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.944271    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c5f62d-44a9-4d43-aff1-6f0191511107-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:08:06.944429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.944286    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:08:06.944429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.944302    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:08:06.945060 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.945039    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-kube-api-access-8ctcf" pod "17c5f62d-44a9-4d43-aff1-6f0191511107" (UID: "17c5f62d-44a9-4d43-aff1-6f0191511107"). InnerVolumeSpecName "kube-api-access-8ctcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:06.945209 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.945192    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-envoy" pod "17c5f62d-44a9-4d43-aff1-6f0191511107" (UID: "17c5f62d-44a9-4d43-aff1-6f0191511107"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:06.945265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.945205    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-token" pod "17c5f62d-44a9-4d43-aff1-6f0191511107" (UID: "17c5f62d-44a9-4d43-aff1-6f0191511107"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:06.945499 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:06.945486    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-podinfo" pod "17c5f62d-44a9-4d43-aff1-6f0191511107" (UID: "17c5f62d-44a9-4d43-aff1-6f0191511107"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:08:07.045467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.045435    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.045467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.045468    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ctcf\" (UniqueName: \"kubernetes.io/projected/17c5f62d-44a9-4d43-aff1-6f0191511107-kube-api-access-8ctcf\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.045834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.045483    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.045834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.045496    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c5f62d-44a9-4d43-aff1-6f0191511107-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:08:07.469032 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.468994    5061 generic.go:356] "Generic (PLEG): container finished" podID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerID="b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93" exitCode=0
Oct 30 03:08:07.469280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.469256    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"
Oct 30 03:08:07.469518 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.469164    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" event={"ID":"17c5f62d-44a9-4d43-aff1-6f0191511107","Type":"ContainerDied","Data":"b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93"}
Oct 30 03:08:07.469623 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.469559    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb" event={"ID":"17c5f62d-44a9-4d43-aff1-6f0191511107","Type":"ContainerDied","Data":"1606eadb52ba0f78b975115af3e81a44ff7e986dec857e383d0aff95a801b08f"}
Oct 30 03:08:07.469623 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.469594    5061 scope.go:122] "RemoveContainer" containerID="b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93"
Oct 30 03:08:07.486330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.486308    5061 scope.go:122] "RemoveContainer" containerID="b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93"
Oct 30 03:08:07.486665 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:08:07.486641    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93\": container with ID starting with b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93 not found: ID does not exist" containerID="b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93"
Oct 30 03:08:07.486811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.486676    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93"} err="failed to get container status \"b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93\": rpc error: code = NotFound desc = could not find container \"b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93\": container with ID starting with b4b1e81160a4ff6bf4fad497ae3b965183c751f47447d914fdb7835ed5307b93 not found: ID does not exist"
Oct 30 03:08:07.489248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.489225    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:07.546373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.546329    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"]
Oct 30 03:08:07.554639 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:07.554610    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-qlwdb"]
Oct 30 03:08:08.452548 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:08.452514    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:08.849979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:08.849944    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" path="/var/lib/kubelet/pods/17c5f62d-44a9-4d43-aff1-6f0191511107/volumes"
Oct 30 03:08:10.812155 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:10.812124    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:13.848052 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:13.848021    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:16.136044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:16.135996    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:18.212976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:18.212935    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:20.608442 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:20.608406    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:22.349384 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:22.349349    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:24.415884 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.415837    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:24.634794 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.634757    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"]
Oct 30 03:08:24.635494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635479    5061 cpu_manager.go:414] "containerMap: removing container" podUID="82ca6e05-6e01-44e5-913a-c285f53af294" containerName="payload-pre-processing"
Oct 30 03:08:24.635548 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635495    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ca6e05-6e01-44e5-913a-c285f53af294" containerName="payload-pre-processing"
Oct 30 03:08:24.635548 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635515    5061 cpu_manager.go:414] "containerMap: removing container" podUID="59d8f59d-ff26-4c89-8c35-896b6bda6b75" containerName="payload-processing"
Oct 30 03:08:24.635548 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635521    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59d8f59d-ff26-4c89-8c35-896b6bda6b75" containerName="payload-processing"
Oct 30 03:08:24.635548 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635533    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d6c421f7-50d6-4401-941d-542df467aa4f" containerName="payload-pre-processing"
Oct 30 03:08:24.635548 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635538    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c421f7-50d6-4401-941d-542df467aa4f" containerName="payload-pre-processing"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635553    5061 cpu_manager.go:414] "containerMap: removing container" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635558    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635569    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1848202f-6bd1-4c94-9e2b-b8627325c908" containerName="payload-processing"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635573    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1848202f-6bd1-4c94-9e2b-b8627325c908" containerName="payload-processing"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635583    5061 cpu_manager.go:414] "containerMap: removing container" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635587    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635601    5061 cpu_manager.go:414] "containerMap: removing container" podUID="311182e9-9db5-4c05-969b-5ea5a94f29f0" containerName="payload-pre-processing"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635605    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311182e9-9db5-4c05-969b-5ea5a94f29f0" containerName="payload-pre-processing"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635612    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e6dcb7ed-b1cc-49d4-a033-eafd0dd90339" containerName="payload-processing"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635616    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6dcb7ed-b1cc-49d4-a033-eafd0dd90339" containerName="payload-processing"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635627    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e9c27684-504b-498c-92f5-8ca35b30b849" containerName="payload-processing"
Oct 30 03:08:24.635705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635631    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c27684-504b-498c-92f5-8ca35b30b849" containerName="payload-processing"
Oct 30 03:08:24.636133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635750    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ca6e05-6e01-44e5-913a-c285f53af294" containerName="payload-pre-processing"
Oct 30 03:08:24.636133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635760    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="59d8f59d-ff26-4c89-8c35-896b6bda6b75" containerName="payload-processing"
Oct 30 03:08:24.636133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635766    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c421f7-50d6-4401-941d-542df467aa4f" containerName="payload-pre-processing"
Oct 30 03:08:24.636133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635774    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c5f62d-44a9-4d43-aff1-6f0191511107" containerName="istio-proxy"
Oct 30 03:08:24.636133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635781    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1848202f-6bd1-4c94-9e2b-b8627325c908" containerName="payload-processing"
Oct 30 03:08:24.636133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635787    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy"
Oct 30 03:08:24.636133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635793    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="311182e9-9db5-4c05-969b-5ea5a94f29f0" containerName="payload-pre-processing"
Oct 30 03:08:24.636133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635800    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6dcb7ed-b1cc-49d4-a033-eafd0dd90339" containerName="payload-processing"
Oct 30 03:08:24.636133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.635808    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c27684-504b-498c-92f5-8ca35b30b849" containerName="payload-processing"
Oct 30 03:08:24.639567 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.639541    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.650837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.650814    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:24.664095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.664075    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"]
Oct 30 03:08:24.664808 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.664783    5061 cpu_manager.go:414] "containerMap: removing container" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy"
Oct 30 03:08:24.664883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.664809    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy"
Oct 30 03:08:24.664953 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.664943    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="059a5e38-5767-417c-93d4-54b7af112f2f" containerName="istio-proxy"
Oct 30 03:08:24.668563 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.668522    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.673561 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.673542    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-be9f2a59-openshift-default-dockercfg-4gctp\""
Oct 30 03:08:24.686233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.686208    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"]
Oct 30 03:08:24.705301 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.705279    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"]
Oct 30 03:08:24.728327 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.728302    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.728483 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.728333    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.728483 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.728372    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.728624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.728485    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.728624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.728540    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.728624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.728579    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.728863 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.728669    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.728863 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.728715    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.728863 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.728748    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.728863 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.728821    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.729094 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.728932    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzcrh\" (UniqueName: \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-kube-api-access-bzcrh\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.729094 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.729006    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.729094 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.729045    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.729094 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.729071    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.729312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.729106    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f43f5c0-8b05-45b2-a0af-ea7852407168-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.729312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.729153    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68h6\" (UniqueName: \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-kube-api-access-h68h6\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.729312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.729204    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.729312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.729284    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.830879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.830808    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.830901    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.830946    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.830980    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831056    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831113    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831150    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831188    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831417 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831228    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831417 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831291    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzcrh\" (UniqueName: \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-kube-api-access-bzcrh\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831417 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831309    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831417 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831337    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831417 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831391    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831652 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831453    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831652 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831486    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831652 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831522    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f43f5c0-8b05-45b2-a0af-ea7852407168-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831652 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831572    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68h6\" (UniqueName: \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-kube-api-access-h68h6\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831652 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831630    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831714    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831737    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831455    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831822    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831838    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.831947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831870    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.831947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.831908    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.832303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.832204    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.832303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.832257    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.832460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.832438    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f43f5c0-8b05-45b2-a0af-ea7852407168-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.834165 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.834141    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.834380 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.834361    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.834482 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.834464    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.834570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.834548    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.850245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.850218    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.850473 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.850449    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzcrh\" (UniqueName: \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-kube-api-access-bzcrh\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.850706 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.850686    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68h6\" (UniqueName: \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-kube-api-access-h68h6\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.851527 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.851508    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:24.955482 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.955422    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:24.983805 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:24.983778    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:25.320544 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:08:25.320511    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f43f5c0_8b05_45b2_a0af_ea7852407168.slice/crio-3a9c3480fd965b1297f390366951c1054a54395ec6b843b28e426202dc0eb916 WatchSource:0}: Error finding container 3a9c3480fd965b1297f390366951c1054a54395ec6b843b28e426202dc0eb916: Status 404 returned error can't find the container with id 3a9c3480fd965b1297f390366951c1054a54395ec6b843b28e426202dc0eb916
Oct 30 03:08:25.323816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.323777    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:08:25.323992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.323881    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:08:25.323992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.323935    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:08:25.331420 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.330290    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"]
Oct 30 03:08:25.357330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.357164    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"]
Oct 30 03:08:25.375073 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:08:25.375042    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61b8339_07eb_4936_91b0_b8b4dd6e7c92.slice/crio-b064d76b3b78c907c6b6912b68668030b69708ca9a6adada58a81f15250c2333 WatchSource:0}: Error finding container b064d76b3b78c907c6b6912b68668030b69708ca9a6adada58a81f15250c2333: Status 404 returned error can't find the container with id b064d76b3b78c907c6b6912b68668030b69708ca9a6adada58a81f15250c2333
Oct 30 03:08:25.378280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.378243    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:08:25.378385 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.378314    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:08:25.378385 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.378352    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:08:25.452593 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:08:25.452542    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:08:25.667752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.667648    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" event={"ID":"d61b8339-07eb-4936-91b0-b8b4dd6e7c92","Type":"ContainerStarted","Data":"10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a"}
Oct 30 03:08:25.667752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.667720    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" event={"ID":"d61b8339-07eb-4936-91b0-b8b4dd6e7c92","Type":"ContainerStarted","Data":"b064d76b3b78c907c6b6912b68668030b69708ca9a6adada58a81f15250c2333"}
Oct 30 03:08:25.670044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.670006    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" event={"ID":"7f43f5c0-8b05-45b2-a0af-ea7852407168","Type":"ContainerStarted","Data":"c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2"}
Oct 30 03:08:25.670190 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.670063    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" event={"ID":"7f43f5c0-8b05-45b2-a0af-ea7852407168","Type":"ContainerStarted","Data":"3a9c3480fd965b1297f390366951c1054a54395ec6b843b28e426202dc0eb916"}
Oct 30 03:08:25.720754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.720675    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" podStartSLOduration=1.7206572599999999 podStartE2EDuration="1.72065726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:25.715770143 +0000 UTC m=+1651.017948508" watchObservedRunningTime="2026-10-30 03:08:25.72065726 +0000 UTC m=+1651.022835626"
Oct 30 03:08:25.794539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.794484    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" podStartSLOduration=1.79446981 podStartE2EDuration="1.79446981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:25.784196892 +0000 UTC m=+1651.086375279" watchObservedRunningTime="2026-10-30 03:08:25.79446981 +0000 UTC m=+1651.096648175"
Oct 30 03:08:25.955577 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.955506    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:25.984885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:25.984833    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:26.960876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:26.959007    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 03:08:26.960876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:26.959082    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 03:08:26.960876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:26.960562    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:27.010875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:27.000998    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:08:27.010875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:27.001077    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 03:08:27.973407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:27.973187    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 03:08:27.973407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:27.973247    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 03:08:27.984124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:27.984092    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:08:27.984252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:27.984129    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 03:08:28.171957 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:28.168412    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:28.840870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:28.835819    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:28.956871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:28.956333    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 03:08:28.956871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:28.956397    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 03:08:28.992168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:28.992132    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:08:28.993021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:28.992977    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 03:08:29.786082 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:29.786007    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:29.983866 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:29.965007    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 03:08:29.983866 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:29.978987    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 03:08:29.983866 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:29.973959    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:29.988534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:29.988501    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:08:29.988828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:29.988777    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 03:08:30.956210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:30.956171    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 03:08:30.956768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:30.956736    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 03:08:30.984315 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:30.984242    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:08:30.984315 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:30.984279    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 03:08:31.055871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:31.046353    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:31.958891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:31.958838    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 03:08:31.959478 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:31.959447    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 03:08:31.985454 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:31.985212    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 03:08:31.985454 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:31.985266    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 03:08:32.936836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:32.936788    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:33.294816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:33.293990    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:33.296963 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:33.296934    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:33.297712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:33.297689    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:08:33.396929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:33.396836    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:33.398418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:33.398321    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:33.400389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:33.399399    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:08:34.680460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:34.680426    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:36.532062 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:36.531970    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:38.200819 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:38.200744    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:39.724720 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.724684    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"]
Oct 30 03:08:39.730122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.730098    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.758013 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.757981    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"]
Oct 30 03:08:39.812410 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.812374    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cvd4\" (UniqueName: \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-kube-api-access-4cvd4\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.812618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.812424    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.812618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.812551    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.812618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.812595    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.812910 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.812676    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.812910 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.812858    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.813019 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.812957    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.813079 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.813016    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.813275 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.813254    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.894470 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.894439    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.915112    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.915184    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.915507    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.915696    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cvd4\" (UniqueName: \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-kube-api-access-4cvd4\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.915774    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.915875    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.915900    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.915926    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-credential-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.915959    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.916166    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.917033    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-socket\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.917247    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-data\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.931283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.918024    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istiod-ca-cert\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.944878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.937879    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-certs\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.982048 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.981759    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cvd4\" (UniqueName: \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-kube-api-access-4cvd4\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.990882 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.988345    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-token\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:39.992723 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:39.991405    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-envoy\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:40.016234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:40.016197    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-podinfo\") pod \"e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") " pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:40.053268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:40.053241    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:40.379956 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:40.379930    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"]
Oct 30 03:08:40.381995 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:08:40.381966    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20e3804_685e_4ee5_beb2_f5a56222a8b8.slice/crio-2aab960e812fda537f7f7b2e14a74ea7d70a427002e632bf39fa5134bd38c4e1 WatchSource:0}: Error finding container 2aab960e812fda537f7f7b2e14a74ea7d70a427002e632bf39fa5134bd38c4e1: Status 404 returned error can't find the container with id 2aab960e812fda537f7f7b2e14a74ea7d70a427002e632bf39fa5134bd38c4e1
Oct 30 03:08:40.384711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:40.384677    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:08:40.384791 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:40.384743    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:08:40.384791 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:40.384775    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:08:40.849434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:40.849403    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" event={"ID":"b20e3804-685e-4ee5-beb2-f5a56222a8b8","Type":"ContainerStarted","Data":"b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524"}
Oct 30 03:08:40.849811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:40.849450    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" event={"ID":"b20e3804-685e-4ee5-beb2-f5a56222a8b8","Type":"ContainerStarted","Data":"2aab960e812fda537f7f7b2e14a74ea7d70a427002e632bf39fa5134bd38c4e1"}
Oct 30 03:08:40.997984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:40.997946    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:41.054030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:41.053998    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:41.055614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:41.055581    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 03:08:41.055793 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:41.055642    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 03:08:41.142927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:41.130070    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:41.313113 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:41.313073    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:42.054870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:42.054427    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:42.055406 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:42.054908    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 03:08:42.055406 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:42.054966    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 03:08:42.232944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:42.232906    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:42.452809 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:42.452712    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:42.873651 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:42.873614    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:43.058286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:43.056257    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 03:08:43.058286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:43.056346    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 03:08:44.054344 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:44.054310    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 03:08:44.054512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:44.054386    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 03:08:44.844289 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:44.844253    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:45.053882 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:45.053831    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 03:08:45.054077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:45.053903    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 03:08:45.816699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:45.816667    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:46.054403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:46.054372    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 03:08:46.054782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:46.054427    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 03:08:46.251708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:46.251669    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:46.684876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:46.673622    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:47.054554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:47.054521    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 03:08:47.055086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:47.054577    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 03:08:47.461319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:47.461215    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:48.059550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:48.059518    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:48.060032 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:48.059974    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:48.060912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:48.060888    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:08:48.061074 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:48.061012    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:48.114100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:48.114038    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" podStartSLOduration=9.114023918000001 podStartE2EDuration="9.114023918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:40.884417504 +0000 UTC m=+1666.186595893" watchObservedRunningTime="2026-10-30 03:08:48.114023918 +0000 UTC m=+1673.416202346"
Oct 30 03:08:49.214703 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:49.214668    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:50.344509 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:50.344475    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:50.491531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:50.491497    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:50.762440 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:50.762356    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:51.096896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:51.086989    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:51.603185 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:51.603136    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:52.538920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:52.538888    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:52.689767 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:52.689735    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:53.582269 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:53.582233    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:55.287983 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:55.287945    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:55.466732 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:08:55.466217    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:08:56.508633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:56.508591    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:56.637095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:56.637065    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:56.764187 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:56.764094    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:57.068091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:57.068061    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:58.299107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:58.299073    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:59.871827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:08:59.871795    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:01.421799 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:01.421762    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:02.132240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:02.130743    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:02.278093 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:02.278049    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:02.427796 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:02.427713    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:03.447917 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:03.447878    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:05.136627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:05.136592    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:05.855868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:05.854197    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:06.033512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:06.033480    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:06.179978 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:06.179798    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:06.595821 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:06.595792    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:08.892996 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:08.892957    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:11.533836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:11.533796    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:12.847466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:12.847430    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:12.981694 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:12.981659    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:13.181539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:13.181463    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"]
Oct 30 03:09:13.181763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:13.181738    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy" containerID="cri-o://5d36d7298164fc92ef58b4639cf7ebd0322293b181e13901a04e7497dcb73b9d" gracePeriod=30
Oct 30 03:09:13.263418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:13.263382    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:13.472020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:13.471926    5061 patch_prober.go:29] interesting pod/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:13.472297 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:13.472268    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:13.649815 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:13.649780    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:14.071372 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:14.071337    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:14.622325 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:14.622290    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:14.833110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:14.828664    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:15.600810 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:15.600771    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:16.679461 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:16.679417    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:17.252679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:17.251008    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:17.425108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:17.425066    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:18.102076 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.038085    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:18.348871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.331359    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:18.397450 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.397413    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" event={"ID":"add485b5-1cb9-4342-8fcc-838fe468aaa7","Type":"ContainerDied","Data":"5d36d7298164fc92ef58b4639cf7ebd0322293b181e13901a04e7497dcb73b9d"}
Oct 30 03:09:18.397610 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.397499    5061 scope.go:122] "RemoveContainer" containerID="c782a9dd35f39524c4e859b9f8f16aebcf3688a00db523b490b3def4406faa69"
Oct 30 03:09:18.397610 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.397411    5061 generic.go:356] "Generic (PLEG): container finished" podID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerID="5d36d7298164fc92ef58b4639cf7ebd0322293b181e13901a04e7497dcb73b9d" exitCode=0
Oct 30 03:09:18.563857 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.563821    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:09:18.648257 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.648172    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-kube-api-access-4cbn6\" (UniqueName: \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-kube-api-access-4cbn6\") pod \"add485b5-1cb9-4342-8fcc-838fe468aaa7\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") "
Oct 30 03:09:18.648423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.648269    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-data\") pod \"add485b5-1cb9-4342-8fcc-838fe468aaa7\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") "
Oct 30 03:09:18.648423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.648315    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add485b5-1cb9-4342-8fcc-838fe468aaa7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add485b5-1cb9-4342-8fcc-838fe468aaa7-istiod-ca-cert\") pod \"add485b5-1cb9-4342-8fcc-838fe468aaa7\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") "
Oct 30 03:09:18.648423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.648377    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-certs\") pod \"add485b5-1cb9-4342-8fcc-838fe468aaa7\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") "
Oct 30 03:09:18.648423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.648400    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-socket\") pod \"add485b5-1cb9-4342-8fcc-838fe468aaa7\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") "
Oct 30 03:09:18.648604 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.648460    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-token\" (UniqueName: \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-token\") pod \"add485b5-1cb9-4342-8fcc-838fe468aaa7\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") "
Oct 30 03:09:18.648720 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.648690    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-data" pod "add485b5-1cb9-4342-8fcc-838fe468aaa7" (UID: "add485b5-1cb9-4342-8fcc-838fe468aaa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:18.648784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.648711    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-certs" pod "add485b5-1cb9-4342-8fcc-838fe468aaa7" (UID: "add485b5-1cb9-4342-8fcc-838fe468aaa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:18.648870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.648777    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add485b5-1cb9-4342-8fcc-838fe468aaa7-istiod-ca-cert" pod "add485b5-1cb9-4342-8fcc-838fe468aaa7" (UID: "add485b5-1cb9-4342-8fcc-838fe468aaa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:18.648935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.648881    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-envoy\") pod \"add485b5-1cb9-4342-8fcc-838fe468aaa7\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") "
Oct 30 03:09:18.649022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.649003    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-socket" pod "add485b5-1cb9-4342-8fcc-838fe468aaa7" (UID: "add485b5-1cb9-4342-8fcc-838fe468aaa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:18.649136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.649118    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-podinfo\") pod \"add485b5-1cb9-4342-8fcc-838fe468aaa7\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") "
Oct 30 03:09:18.649201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.649186    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-credential-socket\") pod \"add485b5-1cb9-4342-8fcc-838fe468aaa7\" (UID: \"add485b5-1cb9-4342-8fcc-838fe468aaa7\") "
Oct 30 03:09:18.649776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.649753    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-credential-socket" pod "add485b5-1cb9-4342-8fcc-838fe468aaa7" (UID: "add485b5-1cb9-4342-8fcc-838fe468aaa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:18.650335 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.650313    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:18.650335 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.650337    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add485b5-1cb9-4342-8fcc-838fe468aaa7-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:18.650533 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.650351    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:18.650533 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.650365    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:18.650533 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.650379    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:18.651324 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.651293    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-token" pod "add485b5-1cb9-4342-8fcc-838fe468aaa7" (UID: "add485b5-1cb9-4342-8fcc-838fe468aaa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:18.651525 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.651505    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-envoy" pod "add485b5-1cb9-4342-8fcc-838fe468aaa7" (UID: "add485b5-1cb9-4342-8fcc-838fe468aaa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:18.651818 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.651796    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-kube-api-access-4cbn6" pod "add485b5-1cb9-4342-8fcc-838fe468aaa7" (UID: "add485b5-1cb9-4342-8fcc-838fe468aaa7"). InnerVolumeSpecName "kube-api-access-4cbn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:18.652993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.652969    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-podinfo" pod "add485b5-1cb9-4342-8fcc-838fe468aaa7" (UID: "add485b5-1cb9-4342-8fcc-838fe468aaa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:18.752401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.752368    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cbn6\" (UniqueName: \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-kube-api-access-4cbn6\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:18.752401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.752396    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:18.752401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.752405    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:18.752624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.752414    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add485b5-1cb9-4342-8fcc-838fe468aaa7-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:18.816103 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:18.816051    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:19.011167 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:19.011074    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:19.410929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:19.410890    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl" event={"ID":"add485b5-1cb9-4342-8fcc-838fe468aaa7","Type":"ContainerDied","Data":"48c00bb8718a5381b21b8b2b57c4f9a2d35a5c830fb583d49160ba34c949af20"}
Oct 30 03:09:19.411440 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:19.410965    5061 scope.go:122] "RemoveContainer" containerID="5d36d7298164fc92ef58b4639cf7ebd0322293b181e13901a04e7497dcb73b9d"
Oct 30 03:09:19.411440 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:19.410910    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"
Oct 30 03:09:19.483956 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:19.483924    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"]
Oct 30 03:09:19.495106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:19.495072    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c1abc2-openshift-default-97b78987f-mq5wl"]
Oct 30 03:09:19.628826 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:19.628775    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:20.426217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:20.426178    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:20.572022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:20.571983    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:20.867666 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:20.867625    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" path="/var/lib/kubelet/pods/add485b5-1cb9-4342-8fcc-838fe468aaa7/volumes"
Oct 30 03:09:21.598991 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:21.598953    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:22.120306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:22.120248    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:22.253696 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:22.253663    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:23.847187 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:23.847157    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:23.983258 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:23.983228    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:24.172918 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:24.150018    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:25.449612 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:09:25.449570    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:09:26.125239 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:26.125199    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:27.688723 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:27.688686    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:29.497365 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:29.497332    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:31.032251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:31.032220    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:32.652032 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:32.652004    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:32.854241 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:32.854167    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:32.980147 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:32.980039    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:34.580071 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:34.580018    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:36.141009 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:36.140977    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:37.995991 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:37.995951    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:38.653908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:38.653873    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:39.498594 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:39.498556    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:40.201872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:40.201627    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:41.632151 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:41.632117    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:43.243994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:43.243956    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:44.576171 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:44.576088    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:45.157656 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:45.157611    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:45.372354 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:45.372092    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:45.492273 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:45.492163    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:45.768106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:45.767998    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:45.893739 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:45.893708    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:46.103175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:46.103143    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:46.147226 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:46.147192    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"]
Oct 30 03:09:46.147492 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:46.147457    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerName="istio-proxy" containerID="cri-o://10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a" gracePeriod=30
Oct 30 03:09:46.174526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:46.174500    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"]
Oct 30 03:09:46.174755 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:46.174731    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerName="istio-proxy" containerID="cri-o://c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2" gracePeriod=30
Oct 30 03:09:46.187118 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:46.187092    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"]
Oct 30 03:09:46.187299 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:46.187281    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy" containerID="cri-o://b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524" gracePeriod=30
Oct 30 03:09:46.329900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:46.326474    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:46.917697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:46.917664    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"]
Oct 30 03:09:46.918223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:46.917979    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy" containerID="cri-o://4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2" gracePeriod=30
Oct 30 03:09:47.055323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:47.055289    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"]
Oct 30 03:09:47.055499 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:47.055478    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy" containerID="cri-o://cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4" gracePeriod=30
Oct 30 03:09:47.426995 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:47.426962    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:47.806908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:47.806873    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"]
Oct 30 03:09:47.807415 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:47.807383    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy" containerID="cri-o://8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8" gracePeriod=30
Oct 30 03:09:48.062168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:48.062092    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:48.062168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:48.062145    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:48.086741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:48.086707    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:48.195972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:48.195936    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:48.298505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:48.298477    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:48.298664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:48.298524    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:48.400341 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:48.400267    5061 patch_prober.go:29] interesting pod/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:09:48.400477 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:48.400328    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:09:49.068186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:49.068150    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:50.780758 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:50.780720    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:51.519114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.518912    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:09:51.583581 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.583530    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:09:51.586712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.586690    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:09:51.657148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.657118    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-podinfo\") pod \"7f43f5c0-8b05-45b2-a0af-ea7852407168\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") "
Oct 30 03:09:51.657386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.657225    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-token\" (UniqueName: \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-token\") pod \"7f43f5c0-8b05-45b2-a0af-ea7852407168\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") "
Oct 30 03:09:51.657386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.657305    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-kube-api-access-bzcrh\" (UniqueName: \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-kube-api-access-bzcrh\") pod \"7f43f5c0-8b05-45b2-a0af-ea7852407168\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") "
Oct 30 03:09:51.657386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.657363    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-data\") pod \"7f43f5c0-8b05-45b2-a0af-ea7852407168\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") "
Oct 30 03:09:51.657568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.657415    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-envoy\") pod \"7f43f5c0-8b05-45b2-a0af-ea7852407168\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") "
Oct 30 03:09:51.657568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.657506    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-socket\") pod \"7f43f5c0-8b05-45b2-a0af-ea7852407168\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") "
Oct 30 03:09:51.657568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.657553    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-credential-socket\") pod \"7f43f5c0-8b05-45b2-a0af-ea7852407168\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") "
Oct 30 03:09:51.657745 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.657648    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-certs\") pod \"7f43f5c0-8b05-45b2-a0af-ea7852407168\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") "
Oct 30 03:09:51.658008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.657829    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f43f5c0-8b05-45b2-a0af-ea7852407168-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f43f5c0-8b05-45b2-a0af-ea7852407168-istiod-ca-cert\") pod \"7f43f5c0-8b05-45b2-a0af-ea7852407168\" (UID: \"7f43f5c0-8b05-45b2-a0af-ea7852407168\") "
Oct 30 03:09:51.659501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.658603    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-certs" pod "7f43f5c0-8b05-45b2-a0af-ea7852407168" (UID: "7f43f5c0-8b05-45b2-a0af-ea7852407168"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.659501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.658642    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-socket" pod "7f43f5c0-8b05-45b2-a0af-ea7852407168" (UID: "7f43f5c0-8b05-45b2-a0af-ea7852407168"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.659501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.659061    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.659501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.659084    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.659501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.659156    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-data" pod "7f43f5c0-8b05-45b2-a0af-ea7852407168" (UID: "7f43f5c0-8b05-45b2-a0af-ea7852407168"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.659501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.659315    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f43f5c0-8b05-45b2-a0af-ea7852407168-istiod-ca-cert" pod "7f43f5c0-8b05-45b2-a0af-ea7852407168" (UID: "7f43f5c0-8b05-45b2-a0af-ea7852407168"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:51.660083 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.660032    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-podinfo" pod "7f43f5c0-8b05-45b2-a0af-ea7852407168" (UID: "7f43f5c0-8b05-45b2-a0af-ea7852407168"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:51.660251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.660170    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-token" pod "7f43f5c0-8b05-45b2-a0af-ea7852407168" (UID: "7f43f5c0-8b05-45b2-a0af-ea7852407168"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.660251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.660223    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-credential-socket" pod "7f43f5c0-8b05-45b2-a0af-ea7852407168" (UID: "7f43f5c0-8b05-45b2-a0af-ea7852407168"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.661322 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.661290    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-kube-api-access-bzcrh" pod "7f43f5c0-8b05-45b2-a0af-ea7852407168" (UID: "7f43f5c0-8b05-45b2-a0af-ea7852407168"). InnerVolumeSpecName "kube-api-access-bzcrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.662618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.662365    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-envoy" pod "7f43f5c0-8b05-45b2-a0af-ea7852407168" (UID: "7f43f5c0-8b05-45b2-a0af-ea7852407168"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.709576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.709481    5061 generic.go:356] "Generic (PLEG): container finished" podID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerID="b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524" exitCode=0
Oct 30 03:09:51.709576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.709557    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"
Oct 30 03:09:51.709794 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.709672    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" event={"ID":"b20e3804-685e-4ee5-beb2-f5a56222a8b8","Type":"ContainerDied","Data":"b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524"}
Oct 30 03:09:51.709794 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.709752    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c" event={"ID":"b20e3804-685e-4ee5-beb2-f5a56222a8b8","Type":"ContainerDied","Data":"2aab960e812fda537f7f7b2e14a74ea7d70a427002e632bf39fa5134bd38c4e1"}
Oct 30 03:09:51.709794 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.709781    5061 scope.go:122] "RemoveContainer" containerID="b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524"
Oct 30 03:09:51.713956 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.713926    5061 generic.go:356] "Generic (PLEG): container finished" podID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerID="10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a" exitCode=0
Oct 30 03:09:51.714078 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.713959    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" event={"ID":"d61b8339-07eb-4936-91b0-b8b4dd6e7c92","Type":"ContainerDied","Data":"10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a"}
Oct 30 03:09:51.714078 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.713989    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"
Oct 30 03:09:51.714078 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.714023    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6" event={"ID":"d61b8339-07eb-4936-91b0-b8b4dd6e7c92","Type":"ContainerDied","Data":"b064d76b3b78c907c6b6912b68668030b69708ca9a6adada58a81f15250c2333"}
Oct 30 03:09:51.717461 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.717434    5061 generic.go:356] "Generic (PLEG): container finished" podID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerID="c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2" exitCode=0
Oct 30 03:09:51.717586 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.717568    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" event={"ID":"7f43f5c0-8b05-45b2-a0af-ea7852407168","Type":"ContainerDied","Data":"c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2"}
Oct 30 03:09:51.717671 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.717609    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v" event={"ID":"7f43f5c0-8b05-45b2-a0af-ea7852407168","Type":"ContainerDied","Data":"3a9c3480fd965b1297f390366951c1054a54395ec6b843b28e426202dc0eb916"}
Oct 30 03:09:51.717735 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.717685    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"
Oct 30 03:09:51.730757 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.730737    5061 scope.go:122] "RemoveContainer" containerID="b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524"
Oct 30 03:09:51.731136 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:09:51.731115    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524\": container with ID starting with b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524 not found: ID does not exist" containerID="b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524"
Oct 30 03:09:51.731201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.731147    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524"} err="failed to get container status \"b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524\": rpc error: code = NotFound desc = could not find container \"b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524\": container with ID starting with b63c36bb15d9ae2745c77524f76d377d2a4591c68b17011730e955bca7af6524 not found: ID does not exist"
Oct 30 03:09:51.731201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.731163    5061 scope.go:122] "RemoveContainer" containerID="10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a"
Oct 30 03:09:51.746073 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.745981    5061 scope.go:122] "RemoveContainer" containerID="10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a"
Oct 30 03:09:51.746501 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:09:51.746458    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a\": container with ID starting with 10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a not found: ID does not exist" containerID="10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a"
Oct 30 03:09:51.746650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.746505    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a"} err="failed to get container status \"10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a\": rpc error: code = NotFound desc = could not find container \"10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a\": container with ID starting with 10ce68d493b30bf322231a7129f247b35be3e08ad21def2f4693bff2d749669a not found: ID does not exist"
Oct 30 03:09:51.746650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.746529    5061 scope.go:122] "RemoveContainer" containerID="c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2"
Oct 30 03:09:51.760159 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760132    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-socket\") pod \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") "
Oct 30 03:09:51.760284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760202    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istiod-ca-cert\") pod \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") "
Oct 30 03:09:51.760284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760234    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-kube-api-access-4cvd4\" (UniqueName: \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-kube-api-access-4cvd4\") pod \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") "
Oct 30 03:09:51.760394 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760323    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-envoy\") pod \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") "
Oct 30 03:09:51.760394 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760358    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-data\") pod \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") "
Oct 30 03:09:51.760506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760391    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-socket" pod "d61b8339-07eb-4936-91b0-b8b4dd6e7c92" (UID: "d61b8339-07eb-4936-91b0-b8b4dd6e7c92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.760506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760411    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-credential-socket\") pod \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") "
Oct 30 03:09:51.760506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760444    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-data\") pod \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") "
Oct 30 03:09:51.760506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760481    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-token\" (UniqueName: \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-token\") pod \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") "
Oct 30 03:09:51.760705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760593    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-envoy\") pod \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") "
Oct 30 03:09:51.760705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760629    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-token\" (UniqueName: \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-token\") pod \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") "
Oct 30 03:09:51.760705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760656    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-podinfo\") pod \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") "
Oct 30 03:09:51.760705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.760692    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-credential-socket" pod "b20e3804-685e-4ee5-beb2-f5a56222a8b8" (UID: "b20e3804-685e-4ee5-beb2-f5a56222a8b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.761045 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.761002    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-data" pod "b20e3804-685e-4ee5-beb2-f5a56222a8b8" (UID: "b20e3804-685e-4ee5-beb2-f5a56222a8b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.761115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.761093    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-data" pod "d61b8339-07eb-4936-91b0-b8b4dd6e7c92" (UID: "d61b8339-07eb-4936-91b0-b8b4dd6e7c92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.761452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.761202    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istiod-ca-cert" pod "d61b8339-07eb-4936-91b0-b8b4dd6e7c92" (UID: "d61b8339-07eb-4936-91b0-b8b4dd6e7c92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:51.761631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.761503    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-credential-socket\") pod \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") "
Oct 30 03:09:51.761631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.761566    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istiod-ca-cert\") pod \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") "
Oct 30 03:09:51.761631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.761609    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-kube-api-access-h68h6\" (UniqueName: \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-kube-api-access-h68h6\") pod \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") "
Oct 30 03:09:51.761822 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.761694    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-certs\") pod \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") "
Oct 30 03:09:51.761822 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.761745    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-podinfo\") pod \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") "
Oct 30 03:09:51.761967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.761869    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-certs\") pod \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\" (UID: \"d61b8339-07eb-4936-91b0-b8b4dd6e7c92\") "
Oct 30 03:09:51.761967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.761907    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-socket\") pod \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\" (UID: \"b20e3804-685e-4ee5-beb2-f5a56222a8b8\") "
Oct 30 03:09:51.762490 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.762246    5061 scope.go:122] "RemoveContainer" containerID="c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2"
Oct 30 03:09:51.762669 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.762580    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-certs" pod "d61b8339-07eb-4936-91b0-b8b4dd6e7c92" (UID: "d61b8339-07eb-4936-91b0-b8b4dd6e7c92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.762669 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.762593    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-credential-socket" pod "d61b8339-07eb-4936-91b0-b8b4dd6e7c92" (UID: "d61b8339-07eb-4936-91b0-b8b4dd6e7c92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.763004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.762980    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istiod-ca-cert" pod "b20e3804-685e-4ee5-beb2-f5a56222a8b8" (UID: "b20e3804-685e-4ee5-beb2-f5a56222a8b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:51.763101 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:09:51.763075    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2\": container with ID starting with c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2 not found: ID does not exist" containerID="c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2"
Oct 30 03:09:51.763169 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.763115    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2"} err="failed to get container status \"c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2\": rpc error: code = NotFound desc = could not find container \"c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2\": container with ID starting with c8e923a1dc60899434833035089cba0c7ac7cc91d491eaf4c67ac3e1f91dc2f2 not found: ID does not exist"
Oct 30 03:09:51.763562 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.763527    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-envoy" pod "d61b8339-07eb-4936-91b0-b8b4dd6e7c92" (UID: "d61b8339-07eb-4936-91b0-b8b4dd6e7c92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.763665 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.763612    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-socket" pod "b20e3804-685e-4ee5-beb2-f5a56222a8b8" (UID: "b20e3804-685e-4ee5-beb2-f5a56222a8b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.763732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.763678    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-certs" pod "b20e3804-685e-4ee5-beb2-f5a56222a8b8" (UID: "b20e3804-685e-4ee5-beb2-f5a56222a8b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.764048 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.763999    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-envoy" pod "b20e3804-685e-4ee5-beb2-f5a56222a8b8" (UID: "b20e3804-685e-4ee5-beb2-f5a56222a8b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:51.764048 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.764021    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-podinfo" pod "b20e3804-685e-4ee5-beb2-f5a56222a8b8" (UID: "b20e3804-685e-4ee5-beb2-f5a56222a8b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:51.764193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.764060    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-token" pod "b20e3804-685e-4ee5-beb2-f5a56222a8b8" (UID: "b20e3804-685e-4ee5-beb2-f5a56222a8b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.764493 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.764475    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.764597 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.764500    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.764597 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.764514    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f43f5c0-8b05-45b2-a0af-ea7852407168-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.764906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.764526    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765013    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765039    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765052    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765065    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765078    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765093    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzcrh\" (UniqueName: \"kubernetes.io/projected/7f43f5c0-8b05-45b2-a0af-ea7852407168-kube-api-access-bzcrh\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765108    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765119    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765129    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765134    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-kube-api-access-4cvd4" pod "b20e3804-685e-4ee5-beb2-f5a56222a8b8" (UID: "b20e3804-685e-4ee5-beb2-f5a56222a8b8"). InnerVolumeSpecName "kube-api-access-4cvd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765142    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765168    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f43f5c0-8b05-45b2-a0af-ea7852407168-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765184    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765200    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765214    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765229    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765242    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765253    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20e3804-685e-4ee5-beb2-f5a56222a8b8-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.765954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765543    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-kube-api-access-h68h6" pod "d61b8339-07eb-4936-91b0-b8b4dd6e7c92" (UID: "d61b8339-07eb-4936-91b0-b8b4dd6e7c92"). InnerVolumeSpecName "kube-api-access-h68h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.765954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765710    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-podinfo" pod "d61b8339-07eb-4936-91b0-b8b4dd6e7c92" (UID: "d61b8339-07eb-4936-91b0-b8b4dd6e7c92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:51.765954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.765813    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-token" pod "d61b8339-07eb-4936-91b0-b8b4dd6e7c92" (UID: "d61b8339-07eb-4936-91b0-b8b4dd6e7c92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:51.792007 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.791799    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"]
Oct 30 03:09:51.799701 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.799674    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-rhg9v"]
Oct 30 03:09:51.866641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.866612    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h68h6\" (UniqueName: \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-kube-api-access-h68h6\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.866641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.866640    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.866827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.866656    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cvd4\" (UniqueName: \"kubernetes.io/projected/b20e3804-685e-4ee5-beb2-f5a56222a8b8-kube-api-access-4cvd4\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:51.866827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:51.866669    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61b8339-07eb-4936-91b0-b8b4dd6e7c92-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.056233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.056201    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"]
Oct 30 03:09:52.060208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.060181    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-6rv2c"]
Oct 30 03:09:52.088207 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.088156    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"]
Oct 30 03:09:52.092308 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.092285    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-be9f2a59-openshift-default-6b7dcc8bb9-4zrn6"]
Oct 30 03:09:52.237803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.237773    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 03:09:52.362861 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.362825    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 03:09:52.372513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.372490    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:52.373674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.373656    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-credential-socket\") pod \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") "
Oct 30 03:09:52.373762 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.373751    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-token\" (UniqueName: \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-token\") pod \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") "
Oct 30 03:09:52.373820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.373807    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-socket\") pod \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") "
Oct 30 03:09:52.373883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.373827    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-certs\") pod \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") "
Oct 30 03:09:52.373948 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.373900    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-credential-socket" pod "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" (UID: "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.373948 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.373922    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istiod-ca-cert\") pod \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") "
Oct 30 03:09:52.374056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.373954    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-envoy\") pod \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") "
Oct 30 03:09:52.374056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.373999    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-data\") pod \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") "
Oct 30 03:09:52.374156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.374100    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-certs" pod "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" (UID: "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.374156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.374117    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-socket" pod "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" (UID: "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.374252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.374155    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-kube-api-access-d5zbt\" (UniqueName: \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-kube-api-access-d5zbt\") pod \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") "
Oct 30 03:09:52.374252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.374200    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-podinfo\") pod \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\" (UID: \"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528\") "
Oct 30 03:09:52.374401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.374368    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istiod-ca-cert" pod "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" (UID: "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:52.374494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.374406    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-data" pod "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" (UID: "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.375071 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.375046    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.375211 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.375075    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.375211 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.375090    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.375211 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.375102    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.375211 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.375114    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.376132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.376107    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-token" pod "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" (UID: "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:52.376303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.376280    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-envoy" pod "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" (UID: "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.376390 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.376371    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-podinfo" pod "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" (UID: "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:52.376602 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.376586    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-kube-api-access-d5zbt" pod "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" (UID: "bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528"). InnerVolumeSpecName "kube-api-access-d5zbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:52.477414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.477374    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-kube-api-access-mrbsk\" (UniqueName: \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-kube-api-access-mrbsk\") pod \"30f8a79d-092c-443e-904a-7de32cd1af90\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") "
Oct 30 03:09:52.477586 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.477529    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-credential-socket\") pod \"30f8a79d-092c-443e-904a-7de32cd1af90\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") "
Oct 30 03:09:52.477644 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.477612    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-data\") pod \"30f8a79d-092c-443e-904a-7de32cd1af90\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") "
Oct 30 03:09:52.477759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.477722    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-envoy\") pod \"30f8a79d-092c-443e-904a-7de32cd1af90\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") "
Oct 30 03:09:52.477821 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.477787    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-certs\") pod \"30f8a79d-092c-443e-904a-7de32cd1af90\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") "
Oct 30 03:09:52.477935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.477917    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-istio-token\" (UniqueName: \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-istio-token\") pod \"30f8a79d-092c-443e-904a-7de32cd1af90\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") "
Oct 30 03:09:52.477999 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.477943    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-credential-socket" pod "30f8a79d-092c-443e-904a-7de32cd1af90" (UID: "30f8a79d-092c-443e-904a-7de32cd1af90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.477999 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.477962    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30f8a79d-092c-443e-904a-7de32cd1af90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f8a79d-092c-443e-904a-7de32cd1af90-istio-podinfo\") pod \"30f8a79d-092c-443e-904a-7de32cd1af90\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") "
Oct 30 03:09:52.478107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.478047    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-socket\") pod \"30f8a79d-092c-443e-904a-7de32cd1af90\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") "
Oct 30 03:09:52.478158 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.478141    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f8a79d-092c-443e-904a-7de32cd1af90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f8a79d-092c-443e-904a-7de32cd1af90-istiod-ca-cert\") pod \"30f8a79d-092c-443e-904a-7de32cd1af90\" (UID: \"30f8a79d-092c-443e-904a-7de32cd1af90\") "
Oct 30 03:09:52.478385 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.478359    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-data" pod "30f8a79d-092c-443e-904a-7de32cd1af90" (UID: "30f8a79d-092c-443e-904a-7de32cd1af90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.479625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.478584    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-certs" pod "30f8a79d-092c-443e-904a-7de32cd1af90" (UID: "30f8a79d-092c-443e-904a-7de32cd1af90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.479625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.479038    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f8a79d-092c-443e-904a-7de32cd1af90-istiod-ca-cert" pod "30f8a79d-092c-443e-904a-7de32cd1af90" (UID: "30f8a79d-092c-443e-904a-7de32cd1af90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:52.479625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.479195    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f8a79d-092c-443e-904a-7de32cd1af90-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.479625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.479219    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.479625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.479234    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.479625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.479247    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.479625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.479263    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5zbt\" (UniqueName: \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-kube-api-access-d5zbt\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.479625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.479277    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.479625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.479290    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.479625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.479303    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.479625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.479262    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-socket" pod "30f8a79d-092c-443e-904a-7de32cd1af90" (UID: "30f8a79d-092c-443e-904a-7de32cd1af90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.481770 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.481744    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30f8a79d-092c-443e-904a-7de32cd1af90-istio-podinfo" pod "30f8a79d-092c-443e-904a-7de32cd1af90" (UID: "30f8a79d-092c-443e-904a-7de32cd1af90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:52.481943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.481919    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-envoy" pod "30f8a79d-092c-443e-904a-7de32cd1af90" (UID: "30f8a79d-092c-443e-904a-7de32cd1af90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:52.483900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.482987    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-kube-api-access-mrbsk" pod "30f8a79d-092c-443e-904a-7de32cd1af90" (UID: "30f8a79d-092c-443e-904a-7de32cd1af90"). InnerVolumeSpecName "kube-api-access-mrbsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:52.483900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.483043    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-istio-token" pod "30f8a79d-092c-443e-904a-7de32cd1af90" (UID: "30f8a79d-092c-443e-904a-7de32cd1af90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:52.580880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.580769    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f8a79d-092c-443e-904a-7de32cd1af90-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.580880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.580807    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.580880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.580821    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrbsk\" (UniqueName: \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-kube-api-access-mrbsk\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.580880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.580833    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f8a79d-092c-443e-904a-7de32cd1af90-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.580880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.580843    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f8a79d-092c-443e-904a-7de32cd1af90-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:52.729123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.729090    5061 generic.go:356] "Generic (PLEG): container finished" podID="30f8a79d-092c-443e-904a-7de32cd1af90" containerID="cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4" exitCode=0
Oct 30 03:09:52.729311 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.729137    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"
Oct 30 03:09:52.729311 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.729176    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" event={"ID":"30f8a79d-092c-443e-904a-7de32cd1af90","Type":"ContainerDied","Data":"cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4"}
Oct 30 03:09:52.729311 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.729235    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl" event={"ID":"30f8a79d-092c-443e-904a-7de32cd1af90","Type":"ContainerDied","Data":"517447bf2c88c68b573b4900652ef6f0d407d7639f5e165790a7805b115f4bfd"}
Oct 30 03:09:52.729311 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.729264    5061 scope.go:122] "RemoveContainer" containerID="cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4"
Oct 30 03:09:52.732317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.732290    5061 generic.go:356] "Generic (PLEG): container finished" podID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerID="4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2" exitCode=0
Oct 30 03:09:52.732464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.732327    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"
Oct 30 03:09:52.732464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.732332    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" event={"ID":"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528","Type":"ContainerDied","Data":"4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2"}
Oct 30 03:09:52.732464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.732454    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5" event={"ID":"bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528","Type":"ContainerDied","Data":"94ea690f253f3eea00f4c6703ebfdbb65110d6b13e9dfe79b16b2ca07ebd6ddf"}
Oct 30 03:09:52.740786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.740769    5061 scope.go:122] "RemoveContainer" containerID="cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4"
Oct 30 03:09:52.741103 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:09:52.741082    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4\": container with ID starting with cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4 not found: ID does not exist" containerID="cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4"
Oct 30 03:09:52.741194 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.741114    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4"} err="failed to get container status \"cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4\": rpc error: code = NotFound desc = could not find container \"cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4\": container with ID starting with cc73f5c588cf77d33bd1393948be1de4234dbcc876b7fc6218c6d713482edae4 not found: ID does not exist"
Oct 30 03:09:52.741194 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.741136    5061 scope.go:122] "RemoveContainer" containerID="4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2"
Oct 30 03:09:52.753798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.753778    5061 scope.go:122] "RemoveContainer" containerID="5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a"
Oct 30 03:09:52.769237 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.769212    5061 scope.go:122] "RemoveContainer" containerID="4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2"
Oct 30 03:09:52.769578 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:09:52.769554    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2\": container with ID starting with 4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2 not found: ID does not exist" containerID="4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2"
Oct 30 03:09:52.769713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.769588    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2"} err="failed to get container status \"4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2\": rpc error: code = NotFound desc = could not find container \"4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2\": container with ID starting with 4a327344b9d0bf6ee9d943a66249dcbe3c455a97035715d59e7a4453e1b9b0e2 not found: ID does not exist"
Oct 30 03:09:52.769713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.769610    5061 scope.go:122] "RemoveContainer" containerID="5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a"
Oct 30 03:09:52.770043 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:09:52.770019    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a\": container with ID starting with 5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a not found: ID does not exist" containerID="5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a"
Oct 30 03:09:52.770130 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.770065    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a"} err="failed to get container status \"5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a\": rpc error: code = NotFound desc = could not find container \"5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a\": container with ID starting with 5fef529000ffc88cc3e8b49da9a9d85ef78ae31a5b4c22498760111f1218c25a not found: ID does not exist"
Oct 30 03:09:52.783863 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.783811    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"]
Oct 30 03:09:52.791771 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.791744    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:52.793029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.793013    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-lsbkl"]
Oct 30 03:09:52.821465 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.821441    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"]
Oct 30 03:09:52.827938 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.827914    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-n2pf5"]
Oct 30 03:09:52.851513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.851486    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" path="/var/lib/kubelet/pods/30f8a79d-092c-443e-904a-7de32cd1af90/volumes"
Oct 30 03:09:52.852281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.852262    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" path="/var/lib/kubelet/pods/7f43f5c0-8b05-45b2-a0af-ea7852407168/volumes"
Oct 30 03:09:52.853012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.852993    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" path="/var/lib/kubelet/pods/b20e3804-685e-4ee5-beb2-f5a56222a8b8/volumes"
Oct 30 03:09:52.854084 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.854061    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" path="/var/lib/kubelet/pods/bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528/volumes"
Oct 30 03:09:52.854555 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.854541    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" path="/var/lib/kubelet/pods/d61b8339-07eb-4936-91b0-b8b4dd6e7c92/volumes"
Oct 30 03:09:52.926646 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:52.926600    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:53.114719 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.114693    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 03:09:53.127511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.127488    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:53.329891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.326378    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-socket\") pod \"5bfa511b-667c-4f96-8509-785691b2cb17\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") "
Oct 30 03:09:53.329891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.326450    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-envoy\") pod \"5bfa511b-667c-4f96-8509-785691b2cb17\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") "
Oct 30 03:09:53.329891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.326488    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bfa511b-667c-4f96-8509-785691b2cb17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfa511b-667c-4f96-8509-785691b2cb17-istio-podinfo\") pod \"5bfa511b-667c-4f96-8509-785691b2cb17\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") "
Oct 30 03:09:53.329891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.326541    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-certs\") pod \"5bfa511b-667c-4f96-8509-785691b2cb17\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") "
Oct 30 03:09:53.329891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.326570    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfa511b-667c-4f96-8509-785691b2cb17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfa511b-667c-4f96-8509-785691b2cb17-istiod-ca-cert\") pod \"5bfa511b-667c-4f96-8509-785691b2cb17\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") "
Oct 30 03:09:53.329891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.326621    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-istio-token\" (UniqueName: \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-istio-token\") pod \"5bfa511b-667c-4f96-8509-785691b2cb17\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") "
Oct 30 03:09:53.329891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.326660    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-credential-socket\") pod \"5bfa511b-667c-4f96-8509-785691b2cb17\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") "
Oct 30 03:09:53.329891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.326755    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-kube-api-access-fqh8s\" (UniqueName: \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-kube-api-access-fqh8s\") pod \"5bfa511b-667c-4f96-8509-785691b2cb17\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") "
Oct 30 03:09:53.329891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.326878    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-data\") pod \"5bfa511b-667c-4f96-8509-785691b2cb17\" (UID: \"5bfa511b-667c-4f96-8509-785691b2cb17\") "
Oct 30 03:09:53.341914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.333984    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-data" pod "5bfa511b-667c-4f96-8509-785691b2cb17" (UID: "5bfa511b-667c-4f96-8509-785691b2cb17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:53.342283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.342208    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-socket" pod "5bfa511b-667c-4f96-8509-785691b2cb17" (UID: "5bfa511b-667c-4f96-8509-785691b2cb17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:53.373436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.358077    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfa511b-667c-4f96-8509-785691b2cb17-istiod-ca-cert" pod "5bfa511b-667c-4f96-8509-785691b2cb17" (UID: "5bfa511b-667c-4f96-8509-785691b2cb17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:09:53.373436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.358365    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-credential-socket" pod "5bfa511b-667c-4f96-8509-785691b2cb17" (UID: "5bfa511b-667c-4f96-8509-785691b2cb17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:53.414902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.409702    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-certs" pod "5bfa511b-667c-4f96-8509-785691b2cb17" (UID: "5bfa511b-667c-4f96-8509-785691b2cb17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:53.441319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.431293    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.441319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.431327    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.441319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.431342    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfa511b-667c-4f96-8509-785691b2cb17-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.441319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.431355    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.441319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.431369    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.441319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.440622    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-kube-api-access-fqh8s" pod "5bfa511b-667c-4f96-8509-785691b2cb17" (UID: "5bfa511b-667c-4f96-8509-785691b2cb17"). InnerVolumeSpecName "kube-api-access-fqh8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:53.441819 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.441657    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bfa511b-667c-4f96-8509-785691b2cb17-istio-podinfo" pod "5bfa511b-667c-4f96-8509-785691b2cb17" (UID: "5bfa511b-667c-4f96-8509-785691b2cb17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:09:53.442043 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.442006    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-istio-token" pod "5bfa511b-667c-4f96-8509-785691b2cb17" (UID: "5bfa511b-667c-4f96-8509-785691b2cb17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:53.445238 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.445206    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-envoy" pod "5bfa511b-667c-4f96-8509-785691b2cb17" (UID: "5bfa511b-667c-4f96-8509-785691b2cb17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:09:53.533330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.533274    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfa511b-667c-4f96-8509-785691b2cb17-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.533330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.533312    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfa511b-667c-4f96-8509-785691b2cb17-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.533330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.533323    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.533330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.533332    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqh8s\" (UniqueName: \"kubernetes.io/projected/5bfa511b-667c-4f96-8509-785691b2cb17-kube-api-access-fqh8s\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:09:53.742898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.742769    5061 generic.go:356] "Generic (PLEG): container finished" podID="5bfa511b-667c-4f96-8509-785691b2cb17" containerID="8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8" exitCode=0
Oct 30 03:09:53.742898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.742822    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"
Oct 30 03:09:53.742898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.742878    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" event={"ID":"5bfa511b-667c-4f96-8509-785691b2cb17","Type":"ContainerDied","Data":"8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8"}
Oct 30 03:09:53.743140 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.742938    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp" event={"ID":"5bfa511b-667c-4f96-8509-785691b2cb17","Type":"ContainerDied","Data":"72848d7d0174b0988162216b96d1d3bc2b462e040a34f09dca0b5003484f7540"}
Oct 30 03:09:53.743140 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.742960    5061 scope.go:122] "RemoveContainer" containerID="8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8"
Oct 30 03:09:53.754137 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.754120    5061 scope.go:122] "RemoveContainer" containerID="8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8"
Oct 30 03:09:53.754385 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:09:53.754367    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8\": container with ID starting with 8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8 not found: ID does not exist" containerID="8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8"
Oct 30 03:09:53.754440 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.754391    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8"} err="failed to get container status \"8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8\": rpc error: code = NotFound desc = could not find container \"8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8\": container with ID starting with 8628605b51cbf5846ec34c3b3b6bfd6364085f761115eb375bfc8b69eef7f8d8 not found: ID does not exist"
Oct 30 03:09:53.796324 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.796297    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"]
Oct 30 03:09:53.805557 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:53.805535    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-s76zp"]
Oct 30 03:09:54.473297 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:54.473262    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:54.849524 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:54.849492    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" path="/var/lib/kubelet/pods/5bfa511b-667c-4f96-8509-785691b2cb17/volumes"
Oct 30 03:09:55.447352 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:09:55.447317    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:09:56.267760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:56.267724    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:58.050685 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:58.050648    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:58.172566 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:58.170920    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:58.299620 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:09:58.299585    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:00.183560 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:00.183528    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:01.843096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:01.843064    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:01.971939 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:01.968445    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:03.896119 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:03.896084    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:04.348232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:04.348200    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:04.465336 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:04.465303    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:05.613164 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:05.613081    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:07.307464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:07.307433    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:08.602051 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:08.602018    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:08.884452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:08.884357    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:10.480977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:10.480944    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:11.014725 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:11.014693    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:11.139551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:11.139519    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:12.150874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:12.147470    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:13.676970 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:13.676930    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:14.944781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:14.944747    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:15.798391 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:15.798336    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:16.097626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:16.097545    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:16.252503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:16.252462    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:16.613190 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:16.613154    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:17.717684 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:17.717652    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:18.020099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:18.020017    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:18.166995 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:18.166958    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:18.719096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:18.711412    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:18.865887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:18.863204    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:19.775828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:19.775794    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:21.549558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:21.549527    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:21.713138 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:10:21.713105    5061 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/NetworkManager-dispatcher.service\": RecentStats: unable to find data in memory cache]"
Oct 30 03:10:22.488334 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:22.488278    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:22.613915 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:22.613883    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:23.762176 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:23.762137    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:23.892873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:23.892753    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:24.193874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:24.182793    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:24.743129 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:24.743090    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:24.862008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:24.861972    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:25.449826 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:10:25.449786    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:10:25.474421 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:25.474384    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:25.631009 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:25.630813    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:25.780953 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:25.768787    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:26.192294 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.192262    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"]
Oct 30 03:10:26.193008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.192992    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerName="istio-proxy"
Oct 30 03:10:26.193008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193009    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerName="istio-proxy"
Oct 30 03:10:26.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193034    5061 cpu_manager.go:414] "containerMap: removing container" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy"
Oct 30 03:10:26.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193039    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy"
Oct 30 03:10:26.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193047    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerName="istio-proxy"
Oct 30 03:10:26.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193053    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerName="istio-proxy"
Oct 30 03:10:26.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193063    5061 cpu_manager.go:414] "containerMap: removing container" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy"
Oct 30 03:10:26.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193067    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy"
Oct 30 03:10:26.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193079    5061 cpu_manager.go:414] "containerMap: removing container" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy"
Oct 30 03:10:26.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193084    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy"
Oct 30 03:10:26.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193095    5061 cpu_manager.go:414] "containerMap: removing container" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy"
Oct 30 03:10:26.193099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193099    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193117    5061 cpu_manager.go:414] "containerMap: removing container" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193122    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193136    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193140    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193248    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20e3804-685e-4ee5-beb2-f5a56222a8b8" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193257    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193265    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61b8339-07eb-4936-91b0-b8b4dd6e7c92" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193274    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f43f5c0-8b05-45b2-a0af-ea7852407168" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193279    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfa511b-667c-4f96-8509-785691b2cb17" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193285    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193292    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="add485b5-1cb9-4342-8fcc-838fe468aaa7" containerName="istio-proxy"
Oct 30 03:10:26.193407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.193299    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f8a79d-092c-443e-904a-7de32cd1af90" containerName="istio-proxy"
Oct 30 03:10:26.197247 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.197220    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:10:26.216809 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.216774    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"]
Oct 30 03:10:26.221706 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.221679    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-537001\""
Oct 30 03:10:26.222068 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.222042    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-537001-dockercfg-pwr5s\""
Oct 30 03:10:26.307776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.307740    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2fc39f-d861-4c39-aa82-fd1317b358ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-537001-66fd974b46-cbngf\" (UID: \"df2fc39f-d861-4c39-aa82-fd1317b358ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:10:26.310939 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.308361    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xxph\" (UniqueName: \"kubernetes.io/projected/df2fc39f-d861-4c39-aa82-fd1317b358ae-kube-api-access-2xxph\") pod \"payload-processing-e2e-inf-b-537001-66fd974b46-cbngf\" (UID: \"df2fc39f-d861-4c39-aa82-fd1317b358ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:10:26.423282 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.423241    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2fc39f-d861-4c39-aa82-fd1317b358ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-537001-66fd974b46-cbngf\" (UID: \"df2fc39f-d861-4c39-aa82-fd1317b358ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:10:26.423645 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.423621    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xxph\" (UniqueName: \"kubernetes.io/projected/df2fc39f-d861-4c39-aa82-fd1317b358ae-kube-api-access-2xxph\") pod \"payload-processing-e2e-inf-b-537001-66fd974b46-cbngf\" (UID: \"df2fc39f-d861-4c39-aa82-fd1317b358ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:10:26.454582 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.454506    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2fc39f-d861-4c39-aa82-fd1317b358ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-537001-66fd974b46-cbngf\" (UID: \"df2fc39f-d861-4c39-aa82-fd1317b358ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:10:26.530959 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.530925    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xxph\" (UniqueName: \"kubernetes.io/projected/df2fc39f-d861-4c39-aa82-fd1317b358ae-kube-api-access-2xxph\") pod \"payload-processing-e2e-inf-b-537001-66fd974b46-cbngf\" (UID: \"df2fc39f-d861-4c39-aa82-fd1317b358ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:10:26.813248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:26.813219    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:10:27.106945 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:27.106838    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:27.123559 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:27.123534    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"]
Oct 30 03:10:27.125091 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:10:27.125065    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2fc39f_d861_4c39_aa82_fd1317b358ae.slice/crio-38c7e0ada430926a992ea0fb6d6a09caf3e53991fa436880d13bda95ff445314 WatchSource:0}: Error finding container 38c7e0ada430926a992ea0fb6d6a09caf3e53991fa436880d13bda95ff445314: Status 404 returned error can't find the container with id 38c7e0ada430926a992ea0fb6d6a09caf3e53991fa436880d13bda95ff445314
Oct 30 03:10:27.215729 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:27.215697    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf" event={"ID":"df2fc39f-d861-4c39-aa82-fd1317b358ae","Type":"ContainerStarted","Data":"aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb"}
Oct 30 03:10:27.216123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:27.215759    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf" event={"ID":"df2fc39f-d861-4c39-aa82-fd1317b358ae","Type":"ContainerStarted","Data":"38c7e0ada430926a992ea0fb6d6a09caf3e53991fa436880d13bda95ff445314"}
Oct 30 03:10:27.216123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:27.215961    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:10:27.254028 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:27.253976    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf" podStartSLOduration=1.253962159 podStartE2EDuration="1.253962159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:27.252223659 +0000 UTC m=+1772.554402024" watchObservedRunningTime="2026-10-30 03:10:27.253962159 +0000 UTC m=+1772.556140525"
Oct 30 03:10:27.455736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:27.455648    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:29.247031 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:29.246993    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:31.311759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:31.311721    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:31.687233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:31.687157    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:31.830875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:31.830524    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:33.319531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:33.319486    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:33.444649 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:33.444613    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:34.127227 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:34.127194    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:36.132139 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:36.132103    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:37.781099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:37.781064    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:38.226081 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:38.226053    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:10:38.923357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:38.923315    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:39.773221 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:39.773188    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:41.767894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:41.767832    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:42.192987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:42.192946    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:43.251974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:43.251943    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:43.369979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:43.369914    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:43.934827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:43.934788    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:45.691098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:45.691058    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:45.820520 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:45.820484    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:46.152130 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:46.152095    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:46.630108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:46.630066    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:47.134428 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:47.134383    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:47.936089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:47.936054    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:48.207517 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:48.207441    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:48.852978 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:48.852946    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:50.059925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.059894    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:50.674229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.674191    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"]
Oct 30 03:10:50.675051 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.675025    5061 cpu_manager.go:414] "containerMap: removing container" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy"
Oct 30 03:10:50.675051 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.675049    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy"
Oct 30 03:10:50.675287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.675272    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7c2d96-e8aa-4ad8-8d93-8d3dbaa82528" containerName="istio-proxy"
Oct 30 03:10:50.679153 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.679135    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.693287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.693267    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cea203-openshift-default-dockercfg-t8d85\""
Oct 30 03:10:50.708210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.708186    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"]
Oct 30 03:10:50.763656 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.763624    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6vvq\" (UniqueName: \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-kube-api-access-q6vvq\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.763823 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.763750    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.763823 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.763789    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-certs\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.763823 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.763805    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-token\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.763963 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.763936    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-podinfo\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.764101 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.764083    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-data\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.764232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.764214    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-credential-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.764325 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.764308    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-envoy\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.764370 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.764356    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070ec01f-42b4-470e-b94b-7fe07c94cd51-istiod-ca-cert\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.837790 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.837763    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:50.865638 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.865610    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-credential-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.865789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.865664    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-envoy\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.865789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.865688    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070ec01f-42b4-470e-b94b-7fe07c94cd51-istiod-ca-cert\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.865789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.865721    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6vvq\" (UniqueName: \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-kube-api-access-q6vvq\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.866012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.865796    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.866012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.865814    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-certs\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.866012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.865829    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-token\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.866012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.865926    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-podinfo\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.866225 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.866030    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-data\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.866225 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.866028    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-credential-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.866225 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.866159    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-certs\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.866225 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.866170    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-socket\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.866471 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.866407    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-data\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.866811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.866789    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070ec01f-42b4-470e-b94b-7fe07c94cd51-istiod-ca-cert\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.867882 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.867843    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-envoy\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.868555 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.868534    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-podinfo\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.883415 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.883385    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-token\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.883678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.883661    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6vvq\" (UniqueName: \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-kube-api-access-q6vvq\") pod \"e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") " pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:50.971990 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.971831    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:50.994105 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:50.994085    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:51.324011 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:51.323985    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"]
Oct 30 03:10:51.325932 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:10:51.325904    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070ec01f_42b4_470e_b94b_7fe07c94cd51.slice/crio-a8f14ae97e29f799b5ec8ddc2b9e3947053db66db9fc14101c4290e14cfb0ac2 WatchSource:0}: Error finding container a8f14ae97e29f799b5ec8ddc2b9e3947053db66db9fc14101c4290e14cfb0ac2: Status 404 returned error can't find the container with id a8f14ae97e29f799b5ec8ddc2b9e3947053db66db9fc14101c4290e14cfb0ac2
Oct 30 03:10:51.328660 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:51.328623    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:10:51.328760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:51.328699    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:10:51.328760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:51.328731    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:10:51.458677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:51.458643    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" event={"ID":"070ec01f-42b4-470e-b94b-7fe07c94cd51","Type":"ContainerStarted","Data":"c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b"}
Oct 30 03:10:51.458833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:51.458724    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" event={"ID":"070ec01f-42b4-470e-b94b-7fe07c94cd51","Type":"ContainerStarted","Data":"a8f14ae97e29f799b5ec8ddc2b9e3947053db66db9fc14101c4290e14cfb0ac2"}
Oct 30 03:10:51.513050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:51.512997    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" podStartSLOduration=1.5129830119999998 podStartE2EDuration="1.512983012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:51.512321447 +0000 UTC m=+1796.814499826" watchObservedRunningTime="2026-10-30 03:10:51.512983012 +0000 UTC m=+1796.815161432"
Oct 30 03:10:51.962558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:51.962526    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:51.995537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:51.995506    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:52.557743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:52.557709    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:52.676002 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:52.675966    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:53.155887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:53.135262    5061 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 03:10:53.155887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:53.135328    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" podUID="070ec01f-42b4-470e-b94b-7fe07c94cd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 03:10:53.907211 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:53.907178    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:53.994959 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:53.994917    5061 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 03:10:53.995148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:53.994985    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" podUID="070ec01f-42b4-470e-b94b-7fe07c94cd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 03:10:54.994509 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:54.994471    5061 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 03:10:54.994935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:54.994520    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" podUID="070ec01f-42b4-470e-b94b-7fe07c94cd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 03:10:55.212657 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:55.212624    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:10:55.213806 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:55.213785    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:10:55.216158 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:55.216141    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:10:55.217031 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:55.217016    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:10:55.473055 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:10:55.473014    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:10:55.986469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:55.986431    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:55.994402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:55.994374    5061 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 03:10:55.994552 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:55.994432    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" podUID="070ec01f-42b4-470e-b94b-7fe07c94cd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 03:10:56.995196 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:56.995164    5061 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 03:10:56.995597 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:56.995223    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" podUID="070ec01f-42b4-470e-b94b-7fe07c94cd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 03:10:57.692220 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:57.692186    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:57.998941 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:57.998867    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:57.999342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:57.999329    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:58.000116 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:58.000091    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:10:58.764136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:58.764101    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:58.878369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:58.878330    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:59.350944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:10:59.350900    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:00.587708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:00.587676    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:00.710879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:00.710826    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:01.270799 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:01.270765    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:02.087115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:02.087075    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"]
Oct 30 03:11:02.087619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:02.087302    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy" containerID="cri-o://126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68" gracePeriod=30
Oct 30 03:11:02.561532 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:02.561499    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:02.675306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:02.675264    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:02.786779 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:02.786749    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"]
Oct 30 03:11:02.786991 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:02.786971    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy" containerID="cri-o://80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7" gracePeriod=30
Oct 30 03:11:03.092066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:03.092032    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:04.870240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:04.870210    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:06.344627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:06.344586    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:06.485278 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:06.480927    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:07.373836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.373813    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 03:11:07.411305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.411240    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-istio-token\" (UniqueName: \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-istio-token\") pod \"48d85b07-775f-4fba-a17e-329a73fec898\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") "
Oct 30 03:11:07.411305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.411299    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d85b07-775f-4fba-a17e-329a73fec898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d85b07-775f-4fba-a17e-329a73fec898-istiod-ca-cert\") pod \"48d85b07-775f-4fba-a17e-329a73fec898\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") "
Oct 30 03:11:07.411490 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.411332    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-kube-api-access-czp2d\" (UniqueName: \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-kube-api-access-czp2d\") pod \"48d85b07-775f-4fba-a17e-329a73fec898\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") "
Oct 30 03:11:07.411490 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.411409    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-credential-socket\") pod \"48d85b07-775f-4fba-a17e-329a73fec898\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") "
Oct 30 03:11:07.411579 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.411542    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-certs\") pod \"48d85b07-775f-4fba-a17e-329a73fec898\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") "
Oct 30 03:11:07.411635 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.411624    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-data\") pod \"48d85b07-775f-4fba-a17e-329a73fec898\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") "
Oct 30 03:11:07.411716 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.411697    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48d85b07-775f-4fba-a17e-329a73fec898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d85b07-775f-4fba-a17e-329a73fec898-istio-podinfo\") pod \"48d85b07-775f-4fba-a17e-329a73fec898\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") "
Oct 30 03:11:07.411772 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.411741    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d85b07-775f-4fba-a17e-329a73fec898-istiod-ca-cert" pod "48d85b07-775f-4fba-a17e-329a73fec898" (UID: "48d85b07-775f-4fba-a17e-329a73fec898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:07.411892 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.411867    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-certs" pod "48d85b07-775f-4fba-a17e-329a73fec898" (UID: "48d85b07-775f-4fba-a17e-329a73fec898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:07.411978 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.411957    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-envoy\") pod \"48d85b07-775f-4fba-a17e-329a73fec898\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") "
Oct 30 03:11:07.412037 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.412023    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-socket\") pod \"48d85b07-775f-4fba-a17e-329a73fec898\" (UID: \"48d85b07-775f-4fba-a17e-329a73fec898\") "
Oct 30 03:11:07.412090 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.412036    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-data" pod "48d85b07-775f-4fba-a17e-329a73fec898" (UID: "48d85b07-775f-4fba-a17e-329a73fec898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:07.412540 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.412519    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-socket" pod "48d85b07-775f-4fba-a17e-329a73fec898" (UID: "48d85b07-775f-4fba-a17e-329a73fec898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:07.412742 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.412722    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-credential-socket" pod "48d85b07-775f-4fba-a17e-329a73fec898" (UID: "48d85b07-775f-4fba-a17e-329a73fec898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:07.412825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.412779    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:07.412825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.412810    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:07.412950 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.412824    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:07.412950 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.412839    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d85b07-775f-4fba-a17e-329a73fec898-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:07.413913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.413886    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-istio-token" pod "48d85b07-775f-4fba-a17e-329a73fec898" (UID: "48d85b07-775f-4fba-a17e-329a73fec898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:07.414207 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.414180    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48d85b07-775f-4fba-a17e-329a73fec898-istio-podinfo" pod "48d85b07-775f-4fba-a17e-329a73fec898" (UID: "48d85b07-775f-4fba-a17e-329a73fec898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:11:07.414207 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.414180    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-kube-api-access-czp2d" pod "48d85b07-775f-4fba-a17e-329a73fec898" (UID: "48d85b07-775f-4fba-a17e-329a73fec898"). InnerVolumeSpecName "kube-api-access-czp2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:07.414418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.414400    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-envoy" pod "48d85b07-775f-4fba-a17e-329a73fec898" (UID: "48d85b07-775f-4fba-a17e-329a73fec898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:07.514568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.514538    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d85b07-775f-4fba-a17e-329a73fec898-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:07.514568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.514566    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:07.514568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.514576    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:07.514813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.514584    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czp2d\" (UniqueName: \"kubernetes.io/projected/48d85b07-775f-4fba-a17e-329a73fec898-kube-api-access-czp2d\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:07.514813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.514594    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d85b07-775f-4fba-a17e-329a73fec898-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:07.603280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.603246    5061 generic.go:356] "Generic (PLEG): container finished" podID="48d85b07-775f-4fba-a17e-329a73fec898" containerID="126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68" exitCode=0
Oct 30 03:11:07.603573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.603342    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"
Oct 30 03:11:07.603785 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.603351    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" event={"ID":"48d85b07-775f-4fba-a17e-329a73fec898","Type":"ContainerDied","Data":"126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68"}
Oct 30 03:11:07.603862 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.603825    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p" event={"ID":"48d85b07-775f-4fba-a17e-329a73fec898","Type":"ContainerDied","Data":"24204e64880e0d794f59c63514e9b896993047f767ee63bbb67de5ccbf824bb5"}
Oct 30 03:11:07.603926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.603890    5061 scope.go:122] "RemoveContainer" containerID="126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68"
Oct 30 03:11:07.625087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.620492    5061 scope.go:122] "RemoveContainer" containerID="126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68"
Oct 30 03:11:07.625087 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:11:07.622090    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68\": container with ID starting with 126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68 not found: ID does not exist" containerID="126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68"
Oct 30 03:11:07.625087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.622136    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68"} err="failed to get container status \"126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68\": rpc error: code = NotFound desc = could not find container \"126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68\": container with ID starting with 126f0200d9b3548812312c7b28aa60a2e0cb3e029913ea6596e59ce1ae7aec68 not found: ID does not exist"
Oct 30 03:11:07.656409 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.656377    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"]
Oct 30 03:11:07.662459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:07.662304    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-dnj4p"]
Oct 30 03:11:08.055030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.055004    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 03:11:08.076301 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.076276    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:08.134576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.132347    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-certs\") pod \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") "
Oct 30 03:11:08.134576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.132961    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-data" pod "7621da70-c0e2-4d2b-983d-1cf05bb7e844" (UID: "7621da70-c0e2-4d2b-983d-1cf05bb7e844"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:08.134576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.133109    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-data\") pod \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") "
Oct 30 03:11:08.134869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.134723    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-envoy\") pod \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") "
Oct 30 03:11:08.136128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.135505    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:08.136128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.135636    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-certs" pod "7621da70-c0e2-4d2b-983d-1cf05bb7e844" (UID: "7621da70-c0e2-4d2b-983d-1cf05bb7e844"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:08.166102 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.166056    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-envoy" pod "7621da70-c0e2-4d2b-983d-1cf05bb7e844" (UID: "7621da70-c0e2-4d2b-983d-1cf05bb7e844"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:08.236534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.236452    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-credential-socket\") pod \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") "
Oct 30 03:11:08.236534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.236528    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-podinfo\") pod \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") "
Oct 30 03:11:08.236755 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.236738    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istiod-ca-cert\") pod \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") "
Oct 30 03:11:08.236802 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.236771    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-token\" (UniqueName: \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-token\") pod \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") "
Oct 30 03:11:08.236842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.236737    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-credential-socket" pod "7621da70-c0e2-4d2b-983d-1cf05bb7e844" (UID: "7621da70-c0e2-4d2b-983d-1cf05bb7e844"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:08.236842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.236868    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-kube-api-access-z6fjs\" (UniqueName: \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-kube-api-access-z6fjs\") pod \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") "
Oct 30 03:11:08.236998 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.236976    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-socket\") pod \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\" (UID: \"7621da70-c0e2-4d2b-983d-1cf05bb7e844\") "
Oct 30 03:11:08.237146 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.237117    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istiod-ca-cert" pod "7621da70-c0e2-4d2b-983d-1cf05bb7e844" (UID: "7621da70-c0e2-4d2b-983d-1cf05bb7e844"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:11:08.237287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.237240    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-socket" pod "7621da70-c0e2-4d2b-983d-1cf05bb7e844" (UID: "7621da70-c0e2-4d2b-983d-1cf05bb7e844"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:08.237898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.237823    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:08.237898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.237891    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:08.238071 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.237909    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:08.238071 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.237924    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:08.238071 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.237941    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:08.238827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.238804    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-podinfo" pod "7621da70-c0e2-4d2b-983d-1cf05bb7e844" (UID: "7621da70-c0e2-4d2b-983d-1cf05bb7e844"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:11:08.239011 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.238960    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-token" pod "7621da70-c0e2-4d2b-983d-1cf05bb7e844" (UID: "7621da70-c0e2-4d2b-983d-1cf05bb7e844"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:08.239067 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.239052    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-kube-api-access-z6fjs" pod "7621da70-c0e2-4d2b-983d-1cf05bb7e844" (UID: "7621da70-c0e2-4d2b-983d-1cf05bb7e844"). InnerVolumeSpecName "kube-api-access-z6fjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:08.339208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.339173    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6fjs\" (UniqueName: \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-kube-api-access-z6fjs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:08.339208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.339205    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:08.339365 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.339218    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7621da70-c0e2-4d2b-983d-1cf05bb7e844-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:11:08.611854 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.611825    5061 generic.go:356] "Generic (PLEG): container finished" podID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerID="80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7" exitCode=0
Oct 30 03:11:08.612335 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.611887    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" event={"ID":"7621da70-c0e2-4d2b-983d-1cf05bb7e844","Type":"ContainerDied","Data":"80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7"}
Oct 30 03:11:08.612335 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.611904    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"
Oct 30 03:11:08.612335 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.611953    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6" event={"ID":"7621da70-c0e2-4d2b-983d-1cf05bb7e844","Type":"ContainerDied","Data":"d44b46987c62cf7920e9b8497b34d17480670cbc69c954e3357f36d77c6ae687"}
Oct 30 03:11:08.612335 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.611987    5061 scope.go:122] "RemoveContainer" containerID="80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7"
Oct 30 03:11:08.624450 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.624434    5061 scope.go:122] "RemoveContainer" containerID="80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7"
Oct 30 03:11:08.624698 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:11:08.624678    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7\": container with ID starting with 80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7 not found: ID does not exist" containerID="80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7"
Oct 30 03:11:08.624777 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.624706    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7"} err="failed to get container status \"80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7\": rpc error: code = NotFound desc = could not find container \"80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7\": container with ID starting with 80f90111062dac3978a8f9b7d703e65dd48b42510bc7b026f8e92852a5d932d7 not found: ID does not exist"
Oct 30 03:11:08.673223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.673200    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"]
Oct 30 03:11:08.683892 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.683860    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-8jqd6"]
Oct 30 03:11:08.849425 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.849395    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d85b07-775f-4fba-a17e-329a73fec898" path="/var/lib/kubelet/pods/48d85b07-775f-4fba-a17e-329a73fec898/volumes"
Oct 30 03:11:08.849905 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:08.849890    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" path="/var/lib/kubelet/pods/7621da70-c0e2-4d2b-983d-1cf05bb7e844/volumes"
Oct 30 03:11:09.786084 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:09.786047    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:10.339999 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:10.339957    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:11.374560 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:11.374523    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:12.936090 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:12.936053    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:14.632325 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.632293    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"]
Oct 30 03:11:14.633019 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.633000    5061 cpu_manager.go:414] "containerMap: removing container" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy"
Oct 30 03:11:14.633092 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.633020    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy"
Oct 30 03:11:14.633092 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.633069    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy"
Oct 30 03:11:14.633092 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.633077    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy"
Oct 30 03:11:14.633203 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.633192    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7621da70-c0e2-4d2b-983d-1cf05bb7e844" containerName="istio-proxy"
Oct 30 03:11:14.633237 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.633209    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d85b07-775f-4fba-a17e-329a73fec898" containerName="istio-proxy"
Oct 30 03:11:14.638489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.638470    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.657016 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.656988    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"]
Oct 30 03:11:14.686593 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.686565    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e7f767-serving-cert\""
Oct 30 03:11:14.733202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.733168    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.733342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.733291    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-metrics-certs\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.733342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.733332    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8741b784-88ae-4230-b4a1-4184f41ab5e1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.733494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.733473    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whkts\" (UniqueName: \"kubernetes.io/projected/8741b784-88ae-4230-b4a1-4184f41ab5e1-kube-api-access-whkts\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.734360 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.734333    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:14.839227 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.839186    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.839543 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.839517    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-metrics-certs\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.840664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.840635    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8741b784-88ae-4230-b4a1-4184f41ab5e1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.840943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.840923    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whkts\" (UniqueName: \"kubernetes.io/projected/8741b784-88ae-4230-b4a1-4184f41ab5e1-kube-api-access-whkts\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.842798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.842770    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8741b784-88ae-4230-b4a1-4184f41ab5e1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.867825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.867784    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:14.875472 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.872183    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-metrics-certs\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.876926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.876304    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whkts\" (UniqueName: \"kubernetes.io/projected/8741b784-88ae-4230-b4a1-4184f41ab5e1-kube-api-access-whkts\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.877769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.877743    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:14.952748 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:14.952665    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:15.263612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:15.263582    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"]
Oct 30 03:11:15.265823 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:11:15.265796    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8741b784_88ae_4230_b4a1_4184f41ab5e1.slice/crio-7118097d3c389334e04b3118c672d74ea94aecda2d0c63e32f9d57cfea6446cd WatchSource:0}: Error finding container 7118097d3c389334e04b3118c672d74ea94aecda2d0c63e32f9d57cfea6446cd: Status 404 returned error can't find the container with id 7118097d3c389334e04b3118c672d74ea94aecda2d0c63e32f9d57cfea6446cd
Oct 30 03:11:15.267677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:15.267656    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:11:15.678353 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:15.678307    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd" event={"ID":"8741b784-88ae-4230-b4a1-4184f41ab5e1","Type":"ContainerStarted","Data":"c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f"}
Oct 30 03:11:15.678353 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:15.678359    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd" event={"ID":"8741b784-88ae-4230-b4a1-4184f41ab5e1","Type":"ContainerStarted","Data":"7118097d3c389334e04b3118c672d74ea94aecda2d0c63e32f9d57cfea6446cd"}
Oct 30 03:11:15.678829 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:15.678577    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:15.731986 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:15.731924    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd" podStartSLOduration=1.4282350990000001 podStartE2EDuration="1.731904451s" totalImagesPullingTime="303.669352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:15.727251831 +0000 UTC m=+1821.029430195" watchObservedRunningTime="2026-10-30 03:11:15.731904451 +0000 UTC m=+1821.034082817"
Oct 30 03:11:16.331943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:16.331908    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:17.249878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:17.249803    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:17.401769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:17.401736    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:17.771186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:17.771150    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:19.577537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:19.577499    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:21.511757 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:21.511719    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:21.696643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:21.696617    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:11:22.100527 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:22.100496    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:22.954059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:22.954027    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:23.150841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:23.150811    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:23.538057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:23.538015    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:25.392460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:25.392426    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:25.497187 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:11:25.497143    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:11:25.587885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:25.587830    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:25.880763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:25.880724    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:26.723607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:26.723558    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:27.366481 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:27.366403    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:27.733979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:27.733883    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:28.050750 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:28.050701    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:29.749389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:29.749338    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:31.844367 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:31.844219    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:32.016814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:32.016724    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:32.241430 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:32.241344    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:33.884995 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:33.884960    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:34.004378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.004344    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"]
Oct 30 03:11:34.010179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.010159    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:11:34.030605 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.030581    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"]
Oct 30 03:11:34.032114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.032093    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gb5x\" (UniqueName: \"kubernetes.io/projected/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-kube-api-access-8gb5x\") pod \"payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh\" (UID: \"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:11:34.032400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.032379    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh\" (UID: \"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:11:34.036055 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.036034    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f21058-dockercfg-sv5d6\""
Oct 30 03:11:34.036160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.036063    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f21058\""
Oct 30 03:11:34.067637 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.067606    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:34.077974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.077950    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"]
Oct 30 03:11:34.083484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.083462    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:11:34.165927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.133720    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh\" (UID: \"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:11:34.169965 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.169924    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r6sk\" (UniqueName: \"kubernetes.io/projected/549d01c2-89c8-4c66-ac68-64857f09098b-kube-api-access-4r6sk\") pod \"payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp\" (UID: \"549d01c2-89c8-4c66-ac68-64857f09098b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:11:34.170313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.170289    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549d01c2-89c8-4c66-ac68-64857f09098b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp\" (UID: \"549d01c2-89c8-4c66-ac68-64857f09098b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:11:34.173980 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.173949    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gb5x\" (UniqueName: \"kubernetes.io/projected/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-kube-api-access-8gb5x\") pod \"payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh\" (UID: \"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:11:34.181974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.136457    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh\" (UID: \"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:11:34.217876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.217709    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"]
Oct 30 03:11:34.279913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.268158    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gb5x\" (UniqueName: \"kubernetes.io/projected/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-kube-api-access-8gb5x\") pod \"payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh\" (UID: \"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:11:34.331648 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.331369    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:11:34.343429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.339142    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6sk\" (UniqueName: \"kubernetes.io/projected/549d01c2-89c8-4c66-ac68-64857f09098b-kube-api-access-4r6sk\") pod \"payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp\" (UID: \"549d01c2-89c8-4c66-ac68-64857f09098b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:11:34.343429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.339287    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549d01c2-89c8-4c66-ac68-64857f09098b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp\" (UID: \"549d01c2-89c8-4c66-ac68-64857f09098b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:11:34.343429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.341359    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549d01c2-89c8-4c66-ac68-64857f09098b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp\" (UID: \"549d01c2-89c8-4c66-ac68-64857f09098b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:11:34.367170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.367137    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r6sk\" (UniqueName: \"kubernetes.io/projected/549d01c2-89c8-4c66-ac68-64857f09098b-kube-api-access-4r6sk\") pod \"payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp\" (UID: \"549d01c2-89c8-4c66-ac68-64857f09098b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:11:34.429209 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.429184    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:11:34.556947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.556865    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:34.680331 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.680303    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"]
Oct 30 03:11:34.681999 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:11:34.681975    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bb52bb_dc5f_46aa_8ebc_24e9ac5fa193.slice/crio-ac9c120e5dc265849a6a3732bfe4ef0a6a493c4348c81c6e1ca4e020c7a1aae1 WatchSource:0}: Error finding container ac9c120e5dc265849a6a3732bfe4ef0a6a493c4348c81c6e1ca4e020c7a1aae1: Status 404 returned error can't find the container with id ac9c120e5dc265849a6a3732bfe4ef0a6a493c4348c81c6e1ca4e020c7a1aae1
Oct 30 03:11:34.751427 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.751346    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:34.785420 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.784278    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"]
Oct 30 03:11:34.787770 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:11:34.787740    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549d01c2_89c8_4c66_ac68_64857f09098b.slice/crio-53e6c19392dc5758fc62cad9d7183a25dfc2afb85fa4bd1501ca6d076e983d5b WatchSource:0}: Error finding container 53e6c19392dc5758fc62cad9d7183a25dfc2afb85fa4bd1501ca6d076e983d5b: Status 404 returned error can't find the container with id 53e6c19392dc5758fc62cad9d7183a25dfc2afb85fa4bd1501ca6d076e983d5b
Oct 30 03:11:34.863266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.862427    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh" event={"ID":"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193","Type":"ContainerStarted","Data":"5c614d637d6525d7cbe8d675cde23ce3a4b458376ef0c04a1223fa3cebeba8c1"}
Oct 30 03:11:34.863266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.862493    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh" event={"ID":"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193","Type":"ContainerStarted","Data":"ac9c120e5dc265849a6a3732bfe4ef0a6a493c4348c81c6e1ca4e020c7a1aae1"}
Oct 30 03:11:34.863266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.862632    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:11:34.864234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.864199    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp" event={"ID":"549d01c2-89c8-4c66-ac68-64857f09098b","Type":"ContainerStarted","Data":"53e6c19392dc5758fc62cad9d7183a25dfc2afb85fa4bd1501ca6d076e983d5b"}
Oct 30 03:11:34.908547 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.908494    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh" podStartSLOduration=1.908480806 podStartE2EDuration="1.908480806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:34.904883666 +0000 UTC m=+1840.207062028" watchObservedRunningTime="2026-10-30 03:11:34.908480806 +0000 UTC m=+1840.210659170"
Oct 30 03:11:34.921346 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:34.921320    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:35.872979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:35.872887    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp" event={"ID":"549d01c2-89c8-4c66-ac68-64857f09098b","Type":"ContainerStarted","Data":"ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90"}
Oct 30 03:11:35.872979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:35.872955    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:11:35.912204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:35.912150    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp" podStartSLOduration=1.912136575 podStartE2EDuration="1.912136575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:35.909977311 +0000 UTC m=+1841.212155676" watchObservedRunningTime="2026-10-30 03:11:35.912136575 +0000 UTC m=+1841.214314964"
Oct 30 03:11:36.858078 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:36.858043    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:39.078192 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:39.078159    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:40.860871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.860704    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:40.871915 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.870485    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"]
Oct 30 03:11:40.876565 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.876469    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:40.902369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.902326    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-9245d1-openshift-default-dockercfg-884td\""
Oct 30 03:11:40.907265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.907238    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"]
Oct 30 03:11:40.950655 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.950620    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:40.950872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.950660    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:40.950872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.950716    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psrqr\" (UniqueName: \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-kube-api-access-psrqr\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:40.950872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.950735    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:40.950872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.950793    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:40.951198 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.950991    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:40.951258 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.951214    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:40.951316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.951303    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:40.951377 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:40.951353    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.052951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.052918    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.052951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.052954    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.053227 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.053019    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrqr\" (UniqueName: \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-kube-api-access-psrqr\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.053227 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.053047    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.053227 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.053107    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.053227 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.053152    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.053460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.053288    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.053460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.053388    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.053460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.053435    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.053875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.053834    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.053969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.053908    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.054155 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.054107    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.054223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.054127    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.054982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.054958    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.056954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.056929    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.057395 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.057372    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.086459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.086390    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.087485 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.087431    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psrqr\" (UniqueName: \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-kube-api-access-psrqr\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.195296 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.195213    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:41.264870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.261400    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:41.553763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.553729    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"]
Oct 30 03:11:41.555746 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:11:41.555721    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c3171d_e418_4df4_b4dc_aa9a8ced8706.slice/crio-7e892e282f0c4603193f1edeb7c9803598635752525f9998b08c2c828825740b WatchSource:0}: Error finding container 7e892e282f0c4603193f1edeb7c9803598635752525f9998b08c2c828825740b: Status 404 returned error can't find the container with id 7e892e282f0c4603193f1edeb7c9803598635752525f9998b08c2c828825740b
Oct 30 03:11:41.558513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.558480    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:41.558624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.558564    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:41.558624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.558609    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:41.921183 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.921079    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" event={"ID":"93c3171d-e418-4df4-b4dc-aa9a8ced8706","Type":"ContainerStarted","Data":"ac9ce0bb35418fc711a4ecc8d049b98780ef38b504c8232f9adddb7416a928d3"}
Oct 30 03:11:41.921183 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:41.921167    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" event={"ID":"93c3171d-e418-4df4-b4dc-aa9a8ced8706","Type":"ContainerStarted","Data":"7e892e282f0c4603193f1edeb7c9803598635752525f9998b08c2c828825740b"}
Oct 30 03:11:42.008674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:42.008593    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" podStartSLOduration=2.008574859 podStartE2EDuration="2.008574859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:41.996896819 +0000 UTC m=+1847.299075184" watchObservedRunningTime="2026-10-30 03:11:42.008574859 +0000 UTC m=+1847.310753225"
Oct 30 03:11:42.195503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:42.195418    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:42.197447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:42.197419    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 03:11:42.197584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:42.197477    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 03:11:43.196355 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:43.196317    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 03:11:43.196903 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:43.196376    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 03:11:43.392714 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:43.391525    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:44.195777 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:44.195743    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 03:11:44.195962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:44.195801    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 03:11:45.048917 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:45.048880    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:45.196638 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:45.196601    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 03:11:45.196834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:45.196688    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 03:11:45.745469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:45.745436    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:45.876599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:45.876568    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:11:45.963775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:45.963719    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:46.195950 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:46.195912    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 03:11:46.196343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:46.195972    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 03:11:46.790417 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:46.790386    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:46.899251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:46.899219    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:11:47.195731 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:47.195698    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 03:11:47.195927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:47.195760    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 03:11:48.196604 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:48.196570    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 03:11:48.197144 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:48.196631    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 03:11:48.321434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:48.321396    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:48.454981 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:48.454835    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:49.213402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:49.213369    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:49.216632 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:49.216606    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:49.217711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:49.217695    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:11:50.167255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:50.167225    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:51.610504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:51.610467    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"]
Oct 30 03:11:51.786300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:51.786269    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:51.955286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:51.955176    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:52.890213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:52.887576    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:54.709104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:54.709063    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:55.447802 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:11:55.447771    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:11:56.611141 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:56.611109    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g" podUID="ccb0f190-1234-4db3-8cc8-e5a969636704" containerName="payload-pre-processing" containerID="cri-o://3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0" gracePeriod=25
Oct 30 03:11:56.839840 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:56.839796    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:57.274147 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:57.274106    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g" podUID="ccb0f190-1234-4db3-8cc8-e5a969636704" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:11:57.408479 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:57.408441    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:58.849837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:58.849806    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:59.637407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:11:59.637368    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:01.147021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:01.146984    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:03.204284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:03.204249    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:04.836101 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:04.836068    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:05.023260 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:05.023226    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:05.266545 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:05.259799    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:06.838937 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:06.838904    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:07.278430 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:07.278337    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g" podUID="ccb0f190-1234-4db3-8cc8-e5a969636704" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:12:07.763081 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:07.763047    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:07.921553 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:07.921515    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:08.085418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:08.085378    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:08.473828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:08.473740    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:08.860480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:08.860445    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:09.335447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:09.335414    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:10.038462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:10.038403    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:10.196870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:10.195156    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:10.869724 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:10.869661    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:11.003204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.003171    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"]
Oct 30 03:12:11.010059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.010006    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.039107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.039075    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"]
Oct 30 03:12:11.096290 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.096255    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.096456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.096389    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.096456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.096423    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.096456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.096449    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.096637 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.096472    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.096637 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.096545    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6fd\" (UniqueName: \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-kube-api-access-rb6fd\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.096749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.096704    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.096842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.096826    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.097022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.097006    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.198346 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.198254    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.198346 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.198314    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.198544 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.198400    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.198544 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.198492    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.198544 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.198525    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.198691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.198555    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.198691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.198590    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.198691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.198628    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6fd\" (UniqueName: \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-kube-api-access-rb6fd\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.198816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.198782    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.198975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.198951    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.199069 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.199044    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.199300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.199271    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.199416 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.199316    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.199506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.199487    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.201077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.201049    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.201166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.201090    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.222571 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.222543    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.223185 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.223162    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6fd\" (UniqueName: \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-kube-api-access-rb6fd\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.337366 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.337340    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:11.885916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.885893    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"]
Oct 30 03:12:11.888183 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:12:11.888144    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bd36df_8df5_4293_9a2a_efa4da29d75d.slice/crio-99619ed436c8865bc2c6275a35097e46bcae55e60a4e330598a79ddb72f982b6 WatchSource:0}: Error finding container 99619ed436c8865bc2c6275a35097e46bcae55e60a4e330598a79ddb72f982b6: Status 404 returned error can't find the container with id 99619ed436c8865bc2c6275a35097e46bcae55e60a4e330598a79ddb72f982b6
Oct 30 03:12:11.891131 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.891096    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:11.891215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.891167    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:11.891215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:11.891199    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:12.264378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:12.264287    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" event={"ID":"a9bd36df-8df5-4293-9a2a-efa4da29d75d","Type":"ContainerStarted","Data":"9a2c597baa8d38e0a1a2f9dab569ac920dacd2ad99f886384578238f9a4d1878"}
Oct 30 03:12:12.264378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:12.264344    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" event={"ID":"a9bd36df-8df5-4293-9a2a-efa4da29d75d","Type":"ContainerStarted","Data":"99619ed436c8865bc2c6275a35097e46bcae55e60a4e330598a79ddb72f982b6"}
Oct 30 03:12:12.318287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:12.318228    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" podStartSLOduration=2.318210965 podStartE2EDuration="2.318210965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:12.317460054 +0000 UTC m=+1877.619638421" watchObservedRunningTime="2026-10-30 03:12:12.318210965 +0000 UTC m=+1877.620389333"
Oct 30 03:12:12.338012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:12.337981    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:12.339969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:12.339943    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 03:12:12.340084 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:12.339988    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 03:12:12.458963 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:12.458926    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:13.298702 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:13.298670    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:13.345974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:13.344829    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 03:12:13.345974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:13.344903    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 03:12:14.342872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:14.341479    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 03:12:14.342872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:14.341539    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 03:12:15.343873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:15.339264    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 03:12:15.343873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:15.339328    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 03:12:15.545418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:15.545377    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:15.885366 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:15.885335    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:16.182976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:16.182834    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:16.341320 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:16.341284    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 03:12:16.341595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:16.341563    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 03:12:16.975160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:16.975129    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:12:17.093703 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.093674    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb0f190-1234-4db3-8cc8-e5a969636704-kube-api-access-5k5l4\" (UniqueName: \"kubernetes.io/projected/ccb0f190-1234-4db3-8cc8-e5a969636704-kube-api-access-5k5l4\") pod \"ccb0f190-1234-4db3-8cc8-e5a969636704\" (UID: \"ccb0f190-1234-4db3-8cc8-e5a969636704\") "
Oct 30 03:12:17.093979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.093965    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb0f190-1234-4db3-8cc8-e5a969636704-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb0f190-1234-4db3-8cc8-e5a969636704-plugins-config-volume\") pod \"ccb0f190-1234-4db3-8cc8-e5a969636704\" (UID: \"ccb0f190-1234-4db3-8cc8-e5a969636704\") "
Oct 30 03:12:17.094367 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.094340    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb0f190-1234-4db3-8cc8-e5a969636704-plugins-config-volume" pod "ccb0f190-1234-4db3-8cc8-e5a969636704" (UID: "ccb0f190-1234-4db3-8cc8-e5a969636704"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:17.094502 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.094487    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb0f190-1234-4db3-8cc8-e5a969636704-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:17.095773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.095750    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb0f190-1234-4db3-8cc8-e5a969636704-kube-api-access-5k5l4" pod "ccb0f190-1234-4db3-8cc8-e5a969636704" (UID: "ccb0f190-1234-4db3-8cc8-e5a969636704"). InnerVolumeSpecName "kube-api-access-5k5l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:17.196415 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.196337    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k5l4\" (UniqueName: \"kubernetes.io/projected/ccb0f190-1234-4db3-8cc8-e5a969636704-kube-api-access-5k5l4\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:17.282179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.282139    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:17.306639 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.306600    5061 generic.go:356] "Generic (PLEG): container finished" podID="ccb0f190-1234-4db3-8cc8-e5a969636704" containerID="3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0" exitCode=0
Oct 30 03:12:17.306923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.306650    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"
Oct 30 03:12:17.306923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.306686    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g" event={"ID":"ccb0f190-1234-4db3-8cc8-e5a969636704","Type":"ContainerDied","Data":"3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0"}
Oct 30 03:12:17.306923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.306752    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g" event={"ID":"ccb0f190-1234-4db3-8cc8-e5a969636704","Type":"ContainerDied","Data":"761ae07c3ccfcf5f2e861bd4e6956381fe96693a654040bcaa9a6ab52d0b3b39"}
Oct 30 03:12:17.306923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.306788    5061 scope.go:122] "RemoveContainer" containerID="3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0"
Oct 30 03:12:17.343966 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.342231    5061 scope.go:122] "RemoveContainer" containerID="3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0"
Oct 30 03:12:17.343966 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.343370    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 03:12:17.343966 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.343442    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 03:12:17.353707 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:12:17.353673    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0\": container with ID starting with 3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0 not found: ID does not exist" containerID="3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0"
Oct 30 03:12:17.354112 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.353718    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0"} err="failed to get container status \"3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0\": rpc error: code = NotFound desc = could not find container \"3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0\": container with ID starting with 3222f1d276c827bd9eb33f301d5f5f6ef2f3ff304e56810167e7a5c4e729a2b0 not found: ID does not exist"
Oct 30 03:12:17.390225 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.390200    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"]
Oct 30 03:12:17.396348 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.396325    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d30e9f-5fbc676779-sxx2g"]
Oct 30 03:12:17.450997 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.450916    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:17.609912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.609842    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:17.907774 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:17.907535    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:18.080893 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.073732    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:18.344664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.339330    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 03:12:18.344664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.339455    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 03:12:18.365285 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.365219    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"]
Oct 30 03:12:18.369900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.369501    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ccb0f190-1234-4db3-8cc8-e5a969636704" containerName="payload-pre-processing"
Oct 30 03:12:18.369900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.369531    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb0f190-1234-4db3-8cc8-e5a969636704" containerName="payload-pre-processing"
Oct 30 03:12:18.461928 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.450061    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb0f190-1234-4db3-8cc8-e5a969636704" containerName="payload-pre-processing"
Oct 30 03:12:18.506867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.489808    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"]
Oct 30 03:12:18.506867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.489949    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:12:18.506867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.505106    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-100ffd\""
Oct 30 03:12:18.528745 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.522704    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d33ad4c-62c1-4403-9fe3-832bf46c5210-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx\" (UID: \"0d33ad4c-62c1-4403-9fe3-832bf46c5210\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:12:18.528745 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.524671    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlsfw\" (UniqueName: \"kubernetes.io/projected/0d33ad4c-62c1-4403-9fe3-832bf46c5210-kube-api-access-hlsfw\") pod \"payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx\" (UID: \"0d33ad4c-62c1-4403-9fe3-832bf46c5210\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:12:18.583965 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.583929    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-100ffd-dockercfg-bcz7r\""
Oct 30 03:12:18.630609 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.630071    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlsfw\" (UniqueName: \"kubernetes.io/projected/0d33ad4c-62c1-4403-9fe3-832bf46c5210-kube-api-access-hlsfw\") pod \"payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx\" (UID: \"0d33ad4c-62c1-4403-9fe3-832bf46c5210\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:12:18.630609 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.630427    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d33ad4c-62c1-4403-9fe3-832bf46c5210-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx\" (UID: \"0d33ad4c-62c1-4403-9fe3-832bf46c5210\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:12:18.631195 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.631172    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d33ad4c-62c1-4403-9fe3-832bf46c5210-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx\" (UID: \"0d33ad4c-62c1-4403-9fe3-832bf46c5210\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:12:18.651979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.651957    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlsfw\" (UniqueName: \"kubernetes.io/projected/0d33ad4c-62c1-4403-9fe3-832bf46c5210-kube-api-access-hlsfw\") pod \"payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx\" (UID: \"0d33ad4c-62c1-4403-9fe3-832bf46c5210\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:12:18.863873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.857636    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:12:18.885109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:18.885030    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb0f190-1234-4db3-8cc8-e5a969636704" path="/var/lib/kubelet/pods/ccb0f190-1234-4db3-8cc8-e5a969636704/volumes"
Oct 30 03:12:19.209428 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:19.209403    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"]
Oct 30 03:12:19.211794 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:12:19.211761    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d33ad4c_62c1_4403_9fe3_832bf46c5210.slice/crio-783d81c7d38c17c81f0bf03d167bbf1940a8e4cf8542b53e9c99861081b42711 WatchSource:0}: Error finding container 783d81c7d38c17c81f0bf03d167bbf1940a8e4cf8542b53e9c99861081b42711: Status 404 returned error can't find the container with id 783d81c7d38c17c81f0bf03d167bbf1940a8e4cf8542b53e9c99861081b42711
Oct 30 03:12:19.343687 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:19.343664    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:19.344089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:19.344072    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:19.344910 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:19.344891    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:12:19.368624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:19.368599    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx" event={"ID":"0d33ad4c-62c1-4403-9fe3-832bf46c5210","Type":"ContainerStarted","Data":"dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7"}
Oct 30 03:12:19.368741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:19.368650    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx" event={"ID":"0d33ad4c-62c1-4403-9fe3-832bf46c5210","Type":"ContainerStarted","Data":"783d81c7d38c17c81f0bf03d167bbf1940a8e4cf8542b53e9c99861081b42711"}
Oct 30 03:12:19.368741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:19.368681    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:12:19.474264 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:19.474164    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx" podStartSLOduration=1.47415006 podStartE2EDuration="1.47415006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:19.473917031 +0000 UTC m=+1884.776095397" watchObservedRunningTime="2026-10-30 03:12:19.47415006 +0000 UTC m=+1884.776328424"
Oct 30 03:12:19.671927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:19.671890    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:22.152473 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:22.152438    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:23.786593 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:23.786557    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:25.342745 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:25.342716    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:25.447787 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:12:25.447756    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:12:25.461740 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:25.461713    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:25.586476 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:25.586442    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:26.045029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.044996    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"]
Oct 30 03:12:26.051337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.051307    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.068073 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.068041    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"]
Oct 30 03:12:26.079471 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.079434    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:26.156395 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.156343    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af43e5ce-2da5-4f93-9296-66e75df36e82-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.156579 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.156436    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.156579 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.156473    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrg6\" (UniqueName: \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-kube-api-access-xdrg6\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.156579 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.156561    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.156743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.156626    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.156743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.156708    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.156868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.156799    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.156987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.156865    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.157060 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.157042    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.217760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.217727    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:26.259214 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.259181    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.259378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.259266    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.259378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.259333    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.259378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.259375    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.259558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.259530    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.259650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.259628    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af43e5ce-2da5-4f93-9296-66e75df36e82-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.259650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.259634    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-credential-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.259764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.259689    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.259764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.259716    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrg6\" (UniqueName: \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-kube-api-access-xdrg6\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.259899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.259758    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-certs\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.259899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.259798    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.260109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.260088    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-socket\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.260176 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.260145    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-data\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.260290 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.260274    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af43e5ce-2da5-4f93-9296-66e75df36e82-istiod-ca-cert\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.262142 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.262117    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-envoy\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.262477 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.262456    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-podinfo\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.280602 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.277345    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-token\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.280602 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.277474    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrg6\" (UniqueName: \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-kube-api-access-xdrg6\") pod \"e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") " pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.374069 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.373993    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:26.849683 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.849660    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"]
Oct 30 03:12:26.851264 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:12:26.851235    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf43e5ce_2da5_4f93_9296_66e75df36e82.slice/crio-58df43aa40d1f8f65dd4618326cb5f512cb6d54582314e7b949617e1467b6053 WatchSource:0}: Error finding container 58df43aa40d1f8f65dd4618326cb5f512cb6d54582314e7b949617e1467b6053: Status 404 returned error can't find the container with id 58df43aa40d1f8f65dd4618326cb5f512cb6d54582314e7b949617e1467b6053
Oct 30 03:12:26.853898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.853865    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:26.853988 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.853927    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:26.853988 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:26.853959    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:27.291955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:27.291866    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:27.469780 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:27.469735    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" event={"ID":"af43e5ce-2da5-4f93-9296-66e75df36e82","Type":"ContainerStarted","Data":"a3ee567cbe37cb91a365b4a79b95f8b79ac44f054b01740e6ccb024d676b193b"}
Oct 30 03:12:27.470304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:27.469797    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" event={"ID":"af43e5ce-2da5-4f93-9296-66e75df36e82","Type":"ContainerStarted","Data":"58df43aa40d1f8f65dd4618326cb5f512cb6d54582314e7b949617e1467b6053"}
Oct 30 03:12:27.542485 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:27.542388    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" podStartSLOduration=1.5423700930000002 podStartE2EDuration="1.542370093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:27.52804466 +0000 UTC m=+1892.830223024" watchObservedRunningTime="2026-10-30 03:12:27.542370093 +0000 UTC m=+1892.844548459"
Oct 30 03:12:28.374361 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:28.374328    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:28.376452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:28.376426    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 03:12:28.376581 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:28.376474    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" podUID="af43e5ce-2da5-4f93-9296-66e75df36e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 03:12:29.045691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:29.035772    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:29.196952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:29.196916    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:29.374456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:29.374364    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 03:12:29.374456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:29.374429    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" podUID="af43e5ce-2da5-4f93-9296-66e75df36e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 03:12:30.375051 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:30.375015    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 03:12:30.375454 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:30.375098    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" podUID="af43e5ce-2da5-4f93-9296-66e75df36e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 03:12:30.378274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:30.378250    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:12:31.080924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:31.080891    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:31.374440 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:31.374361    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 03:12:31.374440 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:31.374418    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" podUID="af43e5ce-2da5-4f93-9296-66e75df36e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 03:12:32.374759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:32.374721    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 03:12:32.375184 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:32.374793    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" podUID="af43e5ce-2da5-4f93-9296-66e75df36e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 03:12:33.374608 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:33.374570    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 03:12:33.374818 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:33.374648    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" podUID="af43e5ce-2da5-4f93-9296-66e75df36e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 03:12:33.390131 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:33.390099    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:33.570210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:33.570172    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:33.860658 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:33.860625    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:34.380364 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:34.380330    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:34.380920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:34.380897    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:34.381629 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:34.381608    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:12:35.216891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:35.216835    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:36.686589 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:36.686554    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:37.692184 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:37.692148    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:38.537178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:38.537133    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:40.292922 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:40.292889    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:41.917907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:41.910548    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:43.077574 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:43.077529    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:43.603618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:43.603590    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:44.295986 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:44.295904    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:45.001085 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:45.001049    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:45.358797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:45.358766    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:45.574837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:45.574802    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:47.327375 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:47.327341    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:47.567109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:47.567077    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"]
Oct 30 03:12:47.567305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:47.567282    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy" containerID="cri-o://43b412aec8a030612c8b3b99ef7643527cf47449cec03fffe5da15d62b20ce06" gracePeriod=30
Oct 30 03:12:47.826623 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:47.826585    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"]
Oct 30 03:12:47.826816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:47.826795    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" podUID="8ab77670-015d-4bbf-97df-75faa1cb0d18" containerName="istio-proxy" containerID="cri-o://562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8" gracePeriod=30
Oct 30 03:12:48.437649 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:48.437619    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:48.471100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:48.471067    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"]
Oct 30 03:12:48.471295 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:48.471274    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy" containerID="cri-o://adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2" gracePeriod=30
Oct 30 03:12:49.076105 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:49.076066    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:50.596732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:50.596703    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:51.637624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:51.637589    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:51.764390 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:51.764359    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:51.963067 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:51.962981    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:52.039960 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:52.039918    5061 patch_prober.go:29] interesting pod/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:12:52.040230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:52.040203    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:12:52.236585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:52.236358    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:52.460592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:52.460559    5061 patch_prober.go:29] interesting pod/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:12:52.460755 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:52.460617    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:12:52.814636 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:52.814603    5061 generic.go:356] "Generic (PLEG): container finished" podID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerID="43b412aec8a030612c8b3b99ef7643527cf47449cec03fffe5da15d62b20ce06" exitCode=0
Oct 30 03:12:52.815120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:52.814677    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" event={"ID":"f71c8959-e21c-4439-9de9-eb48ea50d938","Type":"ContainerDied","Data":"43b412aec8a030612c8b3b99ef7643527cf47449cec03fffe5da15d62b20ce06"}
Oct 30 03:12:52.946842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:52.946820    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 03:12:53.038031 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.037995    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f71c8959-e21c-4439-9de9-eb48ea50d938-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71c8959-e21c-4439-9de9-eb48ea50d938-istiod-ca-cert\") pod \"f71c8959-e21c-4439-9de9-eb48ea50d938\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") "
Oct 30 03:12:53.038206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.038124    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-podinfo\") pod \"f71c8959-e21c-4439-9de9-eb48ea50d938\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") "
Oct 30 03:12:53.038206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.038202    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-certs\") pod \"f71c8959-e21c-4439-9de9-eb48ea50d938\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") "
Oct 30 03:12:53.038340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.038266    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-credential-socket\") pod \"f71c8959-e21c-4439-9de9-eb48ea50d938\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") "
Oct 30 03:12:53.038405 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.038353    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-token\" (UniqueName: \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-token\") pod \"f71c8959-e21c-4439-9de9-eb48ea50d938\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") "
Oct 30 03:12:53.038405 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.038385    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-data\") pod \"f71c8959-e21c-4439-9de9-eb48ea50d938\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") "
Oct 30 03:12:53.038515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.038410    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71c8959-e21c-4439-9de9-eb48ea50d938-istiod-ca-cert" pod "f71c8959-e21c-4439-9de9-eb48ea50d938" (UID: "f71c8959-e21c-4439-9de9-eb48ea50d938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:53.038515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.038472    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-envoy\") pod \"f71c8959-e21c-4439-9de9-eb48ea50d938\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") "
Oct 30 03:12:53.038628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.038540    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-kube-api-access-xlwzh\" (UniqueName: \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-kube-api-access-xlwzh\") pod \"f71c8959-e21c-4439-9de9-eb48ea50d938\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") "
Oct 30 03:12:53.038628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.038616    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-socket\") pod \"f71c8959-e21c-4439-9de9-eb48ea50d938\" (UID: \"f71c8959-e21c-4439-9de9-eb48ea50d938\") "
Oct 30 03:12:53.038736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.038695    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-credential-socket" pod "f71c8959-e21c-4439-9de9-eb48ea50d938" (UID: "f71c8959-e21c-4439-9de9-eb48ea50d938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.039200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.038923    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-certs" pod "f71c8959-e21c-4439-9de9-eb48ea50d938" (UID: "f71c8959-e21c-4439-9de9-eb48ea50d938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.039200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.039014    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-data" pod "f71c8959-e21c-4439-9de9-eb48ea50d938" (UID: "f71c8959-e21c-4439-9de9-eb48ea50d938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.039200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.039165    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-socket" pod "f71c8959-e21c-4439-9de9-eb48ea50d938" (UID: "f71c8959-e21c-4439-9de9-eb48ea50d938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.039573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.039484    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.039573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.039509    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.039573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.039523    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.039573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.039537    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.039573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.039551    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71c8959-e21c-4439-9de9-eb48ea50d938-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.041931 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.041903    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-podinfo" pod "f71c8959-e21c-4439-9de9-eb48ea50d938" (UID: "f71c8959-e21c-4439-9de9-eb48ea50d938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:12:53.043463 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.043437    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-token" pod "f71c8959-e21c-4439-9de9-eb48ea50d938" (UID: "f71c8959-e21c-4439-9de9-eb48ea50d938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:53.043993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.043973    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-kube-api-access-xlwzh" pod "f71c8959-e21c-4439-9de9-eb48ea50d938" (UID: "f71c8959-e21c-4439-9de9-eb48ea50d938"). InnerVolumeSpecName "kube-api-access-xlwzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:53.043993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.043977    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-envoy" pod "f71c8959-e21c-4439-9de9-eb48ea50d938" (UID: "f71c8959-e21c-4439-9de9-eb48ea50d938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.086409 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.086383    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:12:53.140366 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.140288    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-token\" (UniqueName: \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-token\") pod \"8ab77670-015d-4bbf-97df-75faa1cb0d18\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") "
Oct 30 03:12:53.140535 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.140370    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-envoy\") pod \"8ab77670-015d-4bbf-97df-75faa1cb0d18\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") "
Oct 30 03:12:53.140535 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.140409    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-socket\") pod \"8ab77670-015d-4bbf-97df-75faa1cb0d18\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") "
Oct 30 03:12:53.140535 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.140475    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-kube-api-access-fwxwz\" (UniqueName: \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-kube-api-access-fwxwz\") pod \"8ab77670-015d-4bbf-97df-75faa1cb0d18\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") "
Oct 30 03:12:53.140535 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.140498    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-credential-socket\") pod \"8ab77670-015d-4bbf-97df-75faa1cb0d18\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") "
Oct 30 03:12:53.140764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.140550    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-podinfo\") pod \"8ab77670-015d-4bbf-97df-75faa1cb0d18\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") "
Oct 30 03:12:53.140764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.140608    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-certs\") pod \"8ab77670-015d-4bbf-97df-75faa1cb0d18\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") "
Oct 30 03:12:53.140764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.140641    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-data\") pod \"8ab77670-015d-4bbf-97df-75faa1cb0d18\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") "
Oct 30 03:12:53.140764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.140735    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab77670-015d-4bbf-97df-75faa1cb0d18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab77670-015d-4bbf-97df-75faa1cb0d18-istiod-ca-cert\") pod \"8ab77670-015d-4bbf-97df-75faa1cb0d18\" (UID: \"8ab77670-015d-4bbf-97df-75faa1cb0d18\") "
Oct 30 03:12:53.141000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.140781    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-socket" pod "8ab77670-015d-4bbf-97df-75faa1cb0d18" (UID: "8ab77670-015d-4bbf-97df-75faa1cb0d18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.141000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.140798    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-credential-socket" pod "8ab77670-015d-4bbf-97df-75faa1cb0d18" (UID: "8ab77670-015d-4bbf-97df-75faa1cb0d18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.141075 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141000    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-certs" pod "8ab77670-015d-4bbf-97df-75faa1cb0d18" (UID: "8ab77670-015d-4bbf-97df-75faa1cb0d18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.141185 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141166    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-data" pod "8ab77670-015d-4bbf-97df-75faa1cb0d18" (UID: "8ab77670-015d-4bbf-97df-75faa1cb0d18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.141422 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141400    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab77670-015d-4bbf-97df-75faa1cb0d18-istiod-ca-cert" pod "8ab77670-015d-4bbf-97df-75faa1cb0d18" (UID: "8ab77670-015d-4bbf-97df-75faa1cb0d18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:53.141688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141663    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.141688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141689    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.141876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141700    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.141876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141713    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab77670-015d-4bbf-97df-75faa1cb0d18-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.141876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141725    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.141876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141736    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlwzh\" (UniqueName: \"kubernetes.io/projected/f71c8959-e21c-4439-9de9-eb48ea50d938-kube-api-access-xlwzh\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.141876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141749    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.141876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141761    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71c8959-e21c-4439-9de9-eb48ea50d938-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.141876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.141774    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.144501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.144475    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-envoy" pod "8ab77670-015d-4bbf-97df-75faa1cb0d18" (UID: "8ab77670-015d-4bbf-97df-75faa1cb0d18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.144612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.144498    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-podinfo" pod "8ab77670-015d-4bbf-97df-75faa1cb0d18" (UID: "8ab77670-015d-4bbf-97df-75faa1cb0d18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:12:53.144721 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.144700    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-kube-api-access-fwxwz" pod "8ab77670-015d-4bbf-97df-75faa1cb0d18" (UID: "8ab77670-015d-4bbf-97df-75faa1cb0d18"). InnerVolumeSpecName "kube-api-access-fwxwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:53.144895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.144835    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-token" pod "8ab77670-015d-4bbf-97df-75faa1cb0d18" (UID: "8ab77670-015d-4bbf-97df-75faa1cb0d18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:53.247436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.247300    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.247436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.247361    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.247436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.247378    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxwz\" (UniqueName: \"kubernetes.io/projected/8ab77670-015d-4bbf-97df-75faa1cb0d18-kube-api-access-fwxwz\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.247436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.247400    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab77670-015d-4bbf-97df-75faa1cb0d18-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.754785 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.754722    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 03:12:53.826609 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.826571    5061 generic.go:356] "Generic (PLEG): container finished" podID="8ab77670-015d-4bbf-97df-75faa1cb0d18" containerID="562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8" exitCode=0
Oct 30 03:12:53.827089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.826667    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"
Oct 30 03:12:53.827089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.826689    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" event={"ID":"8ab77670-015d-4bbf-97df-75faa1cb0d18","Type":"ContainerDied","Data":"562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8"}
Oct 30 03:12:53.827089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.826797    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z" event={"ID":"8ab77670-015d-4bbf-97df-75faa1cb0d18","Type":"ContainerDied","Data":"1246cc3e14444995ac9440116dbdebe5e1a27f755f472a3b2c2c651817861cab"}
Oct 30 03:12:53.827089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.826827    5061 scope.go:122] "RemoveContainer" containerID="562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8"
Oct 30 03:12:53.831259 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.831222    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv" event={"ID":"f71c8959-e21c-4439-9de9-eb48ea50d938","Type":"ContainerDied","Data":"e57d1da29e74f9e0caad6b43c8fe26a7f4fccf70090005ac7ed86a51a1c05eaa"}
Oct 30 03:12:53.831401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.831350    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"
Oct 30 03:12:53.837304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.837283    5061 generic.go:356] "Generic (PLEG): container finished" podID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerID="adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2" exitCode=0
Oct 30 03:12:53.837474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.837422    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" event={"ID":"f7983d72-b33b-4849-94ac-58c2a0e8fbc7","Type":"ContainerDied","Data":"adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2"}
Oct 30 03:12:53.837474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.837462    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd" event={"ID":"f7983d72-b33b-4849-94ac-58c2a0e8fbc7","Type":"ContainerDied","Data":"3208a8f1cc31f1353de3c84876aafbff6eda0a7cce36eee5806baaacd49220a6"}
Oct 30 03:12:53.837587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.837527    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"
Oct 30 03:12:53.841489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.841471    5061 scope.go:122] "RemoveContainer" containerID="562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8"
Oct 30 03:12:53.841893 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:12:53.841767    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8\": container with ID starting with 562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8 not found: ID does not exist" containerID="562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8"
Oct 30 03:12:53.841893 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.841804    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8"} err="failed to get container status \"562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8\": rpc error: code = NotFound desc = could not find container \"562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8\": container with ID starting with 562a86e919784193cb3cbe5c7964b7a64432ab69c2e0c86a28dd3386dd610cc8 not found: ID does not exist"
Oct 30 03:12:53.841893 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.841823    5061 scope.go:122] "RemoveContainer" containerID="43b412aec8a030612c8b3b99ef7643527cf47449cec03fffe5da15d62b20ce06"
Oct 30 03:12:53.851673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.851652    5061 scope.go:122] "RemoveContainer" containerID="adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2"
Oct 30 03:12:53.855012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.854988    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-kube-api-access-69pvp\" (UniqueName: \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-kube-api-access-69pvp\") pod \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") "
Oct 30 03:12:53.855124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.855046    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-credential-socket\") pod \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") "
Oct 30 03:12:53.855124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.855106    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-token\" (UniqueName: \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-token\") pod \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") "
Oct 30 03:12:53.855249 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.855166    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-envoy\") pod \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") "
Oct 30 03:12:53.855306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.855265    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istiod-ca-cert\") pod \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") "
Oct 30 03:12:53.855306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.855285    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-credential-socket" pod "f7983d72-b33b-4849-94ac-58c2a0e8fbc7" (UID: "f7983d72-b33b-4849-94ac-58c2a0e8fbc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.855414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.855378    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-certs\") pod \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") "
Oct 30 03:12:53.855480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.855444    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-socket\") pod \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") "
Oct 30 03:12:53.855538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.855495    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-podinfo\") pod \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") "
Oct 30 03:12:53.855654 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.855568    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-data\") pod \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\" (UID: \"f7983d72-b33b-4849-94ac-58c2a0e8fbc7\") "
Oct 30 03:12:53.856265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.856238    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.856679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.856458    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-data" pod "f7983d72-b33b-4849-94ac-58c2a0e8fbc7" (UID: "f7983d72-b33b-4849-94ac-58c2a0e8fbc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.856679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.856473    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-socket" pod "f7983d72-b33b-4849-94ac-58c2a0e8fbc7" (UID: "f7983d72-b33b-4849-94ac-58c2a0e8fbc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.856879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.856685    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istiod-ca-cert" pod "f7983d72-b33b-4849-94ac-58c2a0e8fbc7" (UID: "f7983d72-b33b-4849-94ac-58c2a0e8fbc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:53.859279 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.857024    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-certs" pod "f7983d72-b33b-4849-94ac-58c2a0e8fbc7" (UID: "f7983d72-b33b-4849-94ac-58c2a0e8fbc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.859279 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.857746    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-token" pod "f7983d72-b33b-4849-94ac-58c2a0e8fbc7" (UID: "f7983d72-b33b-4849-94ac-58c2a0e8fbc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:53.860005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.859978    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-podinfo" pod "f7983d72-b33b-4849-94ac-58c2a0e8fbc7" (UID: "f7983d72-b33b-4849-94ac-58c2a0e8fbc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:12:53.864252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.864231    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-envoy" pod "f7983d72-b33b-4849-94ac-58c2a0e8fbc7" (UID: "f7983d72-b33b-4849-94ac-58c2a0e8fbc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:53.865431 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.865410    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-kube-api-access-69pvp" pod "f7983d72-b33b-4849-94ac-58c2a0e8fbc7" (UID: "f7983d72-b33b-4849-94ac-58c2a0e8fbc7"). InnerVolumeSpecName "kube-api-access-69pvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:53.866820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.866779    5061 scope.go:122] "RemoveContainer" containerID="adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2"
Oct 30 03:12:53.867129 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:12:53.867107    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2\": container with ID starting with adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2 not found: ID does not exist" containerID="adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2"
Oct 30 03:12:53.867204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.867141    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2"} err="failed to get container status \"adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2\": rpc error: code = NotFound desc = could not find container \"adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2\": container with ID starting with adcb488024dc6968d02390e67eb251a3e800e9cdf60f4e9c688b89bd92ef52a2 not found: ID does not exist"
Oct 30 03:12:53.876917 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.876892    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"]
Oct 30 03:12:53.881177 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.881151    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-sdn9z"]
Oct 30 03:12:53.935266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.935232    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"]
Oct 30 03:12:53.939719 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.939691    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-9wzsv"]
Oct 30 03:12:53.957907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.957877    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.957907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.957907    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.958066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.957920    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.958066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.957935    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.958066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.957947    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69pvp\" (UniqueName: \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-kube-api-access-69pvp\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.958066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.957960    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.958066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.957973    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:53.958066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:53.957985    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7983d72-b33b-4849-94ac-58c2a0e8fbc7-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:12:54.183378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:54.183341    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"]
Oct 30 03:12:54.188007 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:54.187978    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-58tcd"]
Oct 30 03:12:54.379932 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:54.379898    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:54.850635 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:54.850601    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab77670-015d-4bbf-97df-75faa1cb0d18" path="/var/lib/kubelet/pods/8ab77670-015d-4bbf-97df-75faa1cb0d18/volumes"
Oct 30 03:12:54.851359 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:54.851338    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" path="/var/lib/kubelet/pods/f71c8959-e21c-4439-9de9-eb48ea50d938/volumes"
Oct 30 03:12:54.852115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:54.852093    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" path="/var/lib/kubelet/pods/f7983d72-b33b-4849-94ac-58c2a0e8fbc7/volumes"
Oct 30 03:12:55.456193 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:12:55.456153    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:12:56.180761 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:56.180726    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:56.769862 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:56.769818    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:57.884566 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:57.884535    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:59.628598 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:12:59.628503    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:01.210146 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:01.210085    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:02.634722 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.634691    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"]
Oct 30 03:13:02.635540 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.635522    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy"
Oct 30 03:13:02.635588 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.635542    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy"
Oct 30 03:13:02.635588 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.635561    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy"
Oct 30 03:13:02.635588 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.635566    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy"
Oct 30 03:13:02.635588 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.635584    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8ab77670-015d-4bbf-97df-75faa1cb0d18" containerName="istio-proxy"
Oct 30 03:13:02.635588 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.635588    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab77670-015d-4bbf-97df-75faa1cb0d18" containerName="istio-proxy"
Oct 30 03:13:02.635749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.635738    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f71c8959-e21c-4439-9de9-eb48ea50d938" containerName="istio-proxy"
Oct 30 03:13:02.635794 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.635753    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7983d72-b33b-4849-94ac-58c2a0e8fbc7" containerName="istio-proxy"
Oct 30 03:13:02.635794 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.635762    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab77670-015d-4bbf-97df-75faa1cb0d18" containerName="istio-proxy"
Oct 30 03:13:02.641203 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.641186    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.668256 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.668232    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-100ffd-serving-cert\""
Oct 30 03:13:02.674488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.674461    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"]
Oct 30 03:13:02.753209 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.753164    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:02.827025 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.824351    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbpk4\" (UniqueName: \"kubernetes.io/projected/b777b635-e2e8-427d-a843-27851fc6cbf6-kube-api-access-dbpk4\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.827025 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.824502    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-maas-api-tls\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.839876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.835599    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-metrics-certs\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.839876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.838973    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b777b635-e2e8-427d-a843-27851fc6cbf6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.882574 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.882519    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:02.940321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.940238    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-metrics-certs\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.940321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.940287    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b777b635-e2e8-427d-a843-27851fc6cbf6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.940551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.940428    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbpk4\" (UniqueName: \"kubernetes.io/projected/b777b635-e2e8-427d-a843-27851fc6cbf6-kube-api-access-dbpk4\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.940551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.940464    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-maas-api-tls\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.941152 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.941126    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b777b635-e2e8-427d-a843-27851fc6cbf6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.942821 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.942796    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-metrics-certs\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.942972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.942829    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-maas-api-tls\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:02.958610 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:02.958588    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbpk4\" (UniqueName: \"kubernetes.io/projected/b777b635-e2e8-427d-a843-27851fc6cbf6-kube-api-access-dbpk4\") pod \"maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:03.010539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:03.010510    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:03.255486 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:03.255403    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:03.499367 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:03.499327    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:03.559973 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:03.559947    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"]
Oct 30 03:13:03.562935 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:13:03.562904    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb777b635_e2e8_427d_a843_27851fc6cbf6.slice/crio-9536e76c6ad205a0020e41571f974314cc8de4cbf41001abcd04ad0b613ffb42 WatchSource:0}: Error finding container 9536e76c6ad205a0020e41571f974314cc8de4cbf41001abcd04ad0b613ffb42: Status 404 returned error can't find the container with id 9536e76c6ad205a0020e41571f974314cc8de4cbf41001abcd04ad0b613ffb42
Oct 30 03:13:03.636711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:03.636665    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:03.918274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:03.918192    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws" event={"ID":"b777b635-e2e8-427d-a843-27851fc6cbf6","Type":"ContainerStarted","Data":"9536e76c6ad205a0020e41571f974314cc8de4cbf41001abcd04ad0b613ffb42"}
Oct 30 03:13:04.600336 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:04.600303    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:04.926502 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:04.926416    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws" event={"ID":"b777b635-e2e8-427d-a843-27851fc6cbf6","Type":"ContainerStarted","Data":"d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b"}
Oct 30 03:13:04.926924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:04.926634    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:04.968540 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:04.968487    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws" podStartSLOduration=2.458253978 podStartE2EDuration="2.968469805s" totalImagesPullingTime="510.215827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:04.965068234 +0000 UTC m=+1930.267246599" watchObservedRunningTime="2026-10-30 03:13:04.968469805 +0000 UTC m=+1930.270648171"
Oct 30 03:13:06.203842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:06.203810    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:07.888575 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:07.888541    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:09.572291 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:09.572258    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:10.963055 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:10.963022    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:13:11.174281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:11.174251    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:11.737693 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:11.737631    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:13.411652 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:13.411617    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:14.310010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:14.309972    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:14.775374 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:14.775290    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:14.950974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:14.950936    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:15.252211 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:15.252143    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:15.399896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:15.399753    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:16.603528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:16.603483    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:18.284222 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:18.284186    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:19.682786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:19.682756    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:19.825800 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:19.825765    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:20.322802 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:20.322769    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:21.340361 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.340324    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:21.435513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.435471    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"]
Oct 30 03:13:21.440336 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.440311    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:13:21.461904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.461873    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"]
Oct 30 03:13:21.466338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.466314    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2305f4\""
Oct 30 03:13:21.480585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.480551    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:21.481516 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.481470    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n\" (UID: \"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:13:21.481516 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.481511    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc9ww\" (UniqueName: \"kubernetes.io/projected/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-kube-api-access-qc9ww\") pod \"payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n\" (UID: \"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:13:21.491274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.491245    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"]
Oct 30 03:13:21.497311 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.497289    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:13:21.520795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.520755    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"]
Oct 30 03:13:21.521310 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.521270    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2305f4-dockercfg-knx67\""
Oct 30 03:13:21.628921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.614160    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n\" (UID: \"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:13:21.629207 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.629175    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc9ww\" (UniqueName: \"kubernetes.io/projected/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-kube-api-access-qc9ww\") pod \"payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n\" (UID: \"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:13:21.643712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.617812    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n\" (UID: \"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:13:21.721873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.701639    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc9ww\" (UniqueName: \"kubernetes.io/projected/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-kube-api-access-qc9ww\") pod \"payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n\" (UID: \"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:13:21.739203 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.739159    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctmq\" (UniqueName: \"kubernetes.io/projected/60f9435d-51ae-4004-a485-3025d4333bd3-kube-api-access-7ctmq\") pod \"payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8\" (UID: \"60f9435d-51ae-4004-a485-3025d4333bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:13:21.739522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.739504    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f9435d-51ae-4004-a485-3025d4333bd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8\" (UID: \"60f9435d-51ae-4004-a485-3025d4333bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:13:21.763469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.763126    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:13:21.846600 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.846072    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctmq\" (UniqueName: \"kubernetes.io/projected/60f9435d-51ae-4004-a485-3025d4333bd3-kube-api-access-7ctmq\") pod \"payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8\" (UID: \"60f9435d-51ae-4004-a485-3025d4333bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:13:21.846600 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.846147    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f9435d-51ae-4004-a485-3025d4333bd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8\" (UID: \"60f9435d-51ae-4004-a485-3025d4333bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:13:21.847217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.847180    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f9435d-51ae-4004-a485-3025d4333bd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8\" (UID: \"60f9435d-51ae-4004-a485-3025d4333bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:13:21.874465 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:21.874431    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctmq\" (UniqueName: \"kubernetes.io/projected/60f9435d-51ae-4004-a485-3025d4333bd3-kube-api-access-7ctmq\") pod \"payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8\" (UID: \"60f9435d-51ae-4004-a485-3025d4333bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:13:22.133048 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:22.133013    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:13:22.227897 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:22.227402    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"]
Oct 30 03:13:22.232020 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:13:22.230285    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74faa5d_6ae5_42ed_adb2_6b0cc14cfa8b.slice/crio-dcd2a636abb8ca8539a283fe6d25222fac118f7ae817ef6a9ea9544b7ce3fc3f WatchSource:0}: Error finding container dcd2a636abb8ca8539a283fe6d25222fac118f7ae817ef6a9ea9544b7ce3fc3f: Status 404 returned error can't find the container with id dcd2a636abb8ca8539a283fe6d25222fac118f7ae817ef6a9ea9544b7ce3fc3f
Oct 30 03:13:22.394560 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:22.394467    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:22.451883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:22.451815    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"]
Oct 30 03:13:23.093397 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:23.093352    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n" event={"ID":"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b","Type":"ContainerStarted","Data":"ac7c2352a7bfd79abfb791d4647d6cc04c2001bf2002c95ecac8772b361a1137"}
Oct 30 03:13:23.093589 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:23.093431    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:13:23.093589 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:23.093451    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n" event={"ID":"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b","Type":"ContainerStarted","Data":"dcd2a636abb8ca8539a283fe6d25222fac118f7ae817ef6a9ea9544b7ce3fc3f"}
Oct 30 03:13:23.094906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:23.094883    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8" event={"ID":"60f9435d-51ae-4004-a485-3025d4333bd3","Type":"ContainerStarted","Data":"ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95"}
Oct 30 03:13:23.095007 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:23.094923    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8" event={"ID":"60f9435d-51ae-4004-a485-3025d4333bd3","Type":"ContainerStarted","Data":"789097d1c704fc12ab462e7bd5eee8972f171b949756ab0f9bc5a88afa608502"}
Oct 30 03:13:23.095061 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:23.095006    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:13:23.130490 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:23.130435    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n" podStartSLOduration=2.130421021 podStartE2EDuration="2.130421021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:23.127895 +0000 UTC m=+1948.430073364" watchObservedRunningTime="2026-10-30 03:13:23.130421021 +0000 UTC m=+1948.432599400"
Oct 30 03:13:23.163628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:23.163510    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8" podStartSLOduration=2.163493912 podStartE2EDuration="2.163493912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:23.162363148 +0000 UTC m=+1948.464541514" watchObservedRunningTime="2026-10-30 03:13:23.163493912 +0000 UTC m=+1948.465672277"
Oct 30 03:13:24.021869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:24.021811    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:25.447927 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:13:25.447889    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:13:25.641982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:25.641948    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:27.317879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:27.317485    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:29.027773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:29.027731    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:30.955348 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:30.955312    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:31.917330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:31.917298    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:32.036859 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:32.036822    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:32.171907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:32.171806    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:32.575384 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:32.575354    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:34.107704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:34.107674    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:13:34.108250 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:34.107744    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:13:34.109195 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:34.109160    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:35.693025 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:35.692978    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:37.272965 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:37.272931    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:38.445115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:38.445084    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:38.569226 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:38.569193    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:38.725723 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:38.725638    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"]
Oct 30 03:13:38.726018 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:38.725898    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" podUID="070ec01f-42b4-470e-b94b-7fe07c94cd51" containerName="istio-proxy" containerID="cri-o://c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b" gracePeriod=30
Oct 30 03:13:38.775488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:38.775461    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:38.891291 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:38.891253    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:40.581414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:40.581380    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:40.700582 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:40.700550    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:42.308300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:42.308266    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:43.001223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:43.001187    5061 patch_prober.go:29] interesting pod/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:13:43.001392 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:43.001242    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" podUID="070ec01f-42b4-470e-b94b-7fe07c94cd51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:13:43.996059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:43.995998    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:44.011419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.011396    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:13:44.102491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.102449    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-envoy\") pod \"070ec01f-42b4-470e-b94b-7fe07c94cd51\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") "
Oct 30 03:13:44.102798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.102495    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-podinfo\") pod \"070ec01f-42b4-470e-b94b-7fe07c94cd51\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") "
Oct 30 03:13:44.102798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.102592    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-credential-socket\") pod \"070ec01f-42b4-470e-b94b-7fe07c94cd51\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") "
Oct 30 03:13:44.102798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.102630    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-kube-api-access-q6vvq\" (UniqueName: \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-kube-api-access-q6vvq\") pod \"070ec01f-42b4-470e-b94b-7fe07c94cd51\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") "
Oct 30 03:13:44.102798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.102677    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-socket\") pod \"070ec01f-42b4-470e-b94b-7fe07c94cd51\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") "
Oct 30 03:13:44.102798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.102759    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-certs\") pod \"070ec01f-42b4-470e-b94b-7fe07c94cd51\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") "
Oct 30 03:13:44.103175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.102802    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-data\") pod \"070ec01f-42b4-470e-b94b-7fe07c94cd51\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") "
Oct 30 03:13:44.103175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.102936    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-token\" (UniqueName: \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-token\") pod \"070ec01f-42b4-470e-b94b-7fe07c94cd51\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") "
Oct 30 03:13:44.103175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.102974    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070ec01f-42b4-470e-b94b-7fe07c94cd51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070ec01f-42b4-470e-b94b-7fe07c94cd51-istiod-ca-cert\") pod \"070ec01f-42b4-470e-b94b-7fe07c94cd51\" (UID: \"070ec01f-42b4-470e-b94b-7fe07c94cd51\") "
Oct 30 03:13:44.105464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.103608    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-certs" pod "070ec01f-42b4-470e-b94b-7fe07c94cd51" (UID: "070ec01f-42b4-470e-b94b-7fe07c94cd51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:44.105464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.103901    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-credential-socket" pod "070ec01f-42b4-470e-b94b-7fe07c94cd51" (UID: "070ec01f-42b4-470e-b94b-7fe07c94cd51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:44.105464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.103967    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-socket" pod "070ec01f-42b4-470e-b94b-7fe07c94cd51" (UID: "070ec01f-42b4-470e-b94b-7fe07c94cd51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:44.105464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.105406    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070ec01f-42b4-470e-b94b-7fe07c94cd51-istiod-ca-cert" pod "070ec01f-42b4-470e-b94b-7fe07c94cd51" (UID: "070ec01f-42b4-470e-b94b-7fe07c94cd51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:44.105919 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.105880    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-data" pod "070ec01f-42b4-470e-b94b-7fe07c94cd51" (UID: "070ec01f-42b4-470e-b94b-7fe07c94cd51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:44.106199 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.106166    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-kube-api-access-q6vvq" pod "070ec01f-42b4-470e-b94b-7fe07c94cd51" (UID: "070ec01f-42b4-470e-b94b-7fe07c94cd51"). InnerVolumeSpecName "kube-api-access-q6vvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:44.118166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.118126    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-envoy" pod "070ec01f-42b4-470e-b94b-7fe07c94cd51" (UID: "070ec01f-42b4-470e-b94b-7fe07c94cd51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:13:44.119557 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.119178    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-token" pod "070ec01f-42b4-470e-b94b-7fe07c94cd51" (UID: "070ec01f-42b4-470e-b94b-7fe07c94cd51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:44.120916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.120573    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-podinfo" pod "070ec01f-42b4-470e-b94b-7fe07c94cd51" (UID: "070ec01f-42b4-470e-b94b-7fe07c94cd51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:13:44.205245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.205215    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070ec01f-42b4-470e-b94b-7fe07c94cd51-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.205245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.205242    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.205245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.205251    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.205462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.205259    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.205462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.205269    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6vvq\" (UniqueName: \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-kube-api-access-q6vvq\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.205462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.205279    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.205462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.205287    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.205462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.205295    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.205462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.205302    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070ec01f-42b4-470e-b94b-7fe07c94cd51-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:13:44.408132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.408099    5061 generic.go:356] "Generic (PLEG): container finished" podID="070ec01f-42b4-470e-b94b-7fe07c94cd51" containerID="c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b" exitCode=0
Oct 30 03:13:44.408328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.408160    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"
Oct 30 03:13:44.408328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.408271    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" event={"ID":"070ec01f-42b4-470e-b94b-7fe07c94cd51","Type":"ContainerDied","Data":"c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b"}
Oct 30 03:13:44.408444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.408338    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn" event={"ID":"070ec01f-42b4-470e-b94b-7fe07c94cd51","Type":"ContainerDied","Data":"a8f14ae97e29f799b5ec8ddc2b9e3947053db66db9fc14101c4290e14cfb0ac2"}
Oct 30 03:13:44.408444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.408369    5061 scope.go:122] "RemoveContainer" containerID="c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b"
Oct 30 03:13:44.421797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.421779    5061 scope.go:122] "RemoveContainer" containerID="c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b"
Oct 30 03:13:44.422130 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:13:44.422104    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b\": container with ID starting with c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b not found: ID does not exist" containerID="c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b"
Oct 30 03:13:44.422230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.422136    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b"} err="failed to get container status \"c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b\": rpc error: code = NotFound desc = could not find container \"c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b\": container with ID starting with c0be338ae234a7895943cb7b7c1d76b5c5949437583faea04637fa203a7d2e3b not found: ID does not exist"
Oct 30 03:13:44.457002 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.456975    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"]
Oct 30 03:13:44.466646 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.466623    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cea203-openshift-default-68cc5c94f7-gwvtn"]
Oct 30 03:13:44.849173 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:44.849140    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070ec01f-42b4-470e-b94b-7fe07c94cd51" path="/var/lib/kubelet/pods/070ec01f-42b4-470e-b94b-7fe07c94cd51/volumes"
Oct 30 03:13:45.445828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.445800    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"]
Oct 30 03:13:45.446558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.446543    5061 cpu_manager.go:414] "containerMap: removing container" podUID="070ec01f-42b4-470e-b94b-7fe07c94cd51" containerName="istio-proxy"
Oct 30 03:13:45.446624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.446559    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070ec01f-42b4-470e-b94b-7fe07c94cd51" containerName="istio-proxy"
Oct 30 03:13:45.446713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.446704    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="070ec01f-42b4-470e-b94b-7fe07c94cd51" containerName="istio-proxy"
Oct 30 03:13:45.451929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.451912    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.467253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.467234    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c1fce3-openshift-default-dockercfg-l84qc\""
Oct 30 03:13:45.482216 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.482188    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"]
Oct 30 03:13:45.579883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.535883    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm27c\" (UniqueName: \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-kube-api-access-xm27c\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.600884 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.580223    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-certs\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.625869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.618508    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-credential-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.625869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.618591    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-token\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.625869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.618645    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istiod-ca-cert\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.625869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.618674    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.625869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.618753    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-podinfo\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.625869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.618816    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-data\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.625869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.618861    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-envoy\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.722952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.721314    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-podinfo\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.722952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.721429    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-data\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.722952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.721460    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-envoy\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.722952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.721672    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm27c\" (UniqueName: \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-kube-api-access-xm27c\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.722952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.721744    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-certs\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.722952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.721805    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-credential-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.722952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.721876    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-token\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.722952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.721928    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istiod-ca-cert\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.722952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.721958    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.722952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.722459    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.724219 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.724192    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-certs\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.742872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.724916    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-credential-socket\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.742872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.739996    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-data\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.742872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.740355    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-envoy\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.742872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.740468    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istiod-ca-cert\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.742872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.741049    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-podinfo\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.746157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.746107    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-token\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.746390 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.746365    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm27c\" (UniqueName: \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-kube-api-access-xm27c\") pod \"e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") " pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:45.766262 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:45.766235    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:46.088784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:46.088746    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:46.097427 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:46.097404    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"]
Oct 30 03:13:46.100308 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:13:46.100284    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ba2b21_c794_4a5b_8c5e_b9ac006f1840.slice/crio-e71fd9f302bb030bc68e8dc7ffeccb65dff4cb5e194d804b4d22713afab0de7e WatchSource:0}: Error finding container e71fd9f302bb030bc68e8dc7ffeccb65dff4cb5e194d804b4d22713afab0de7e: Status 404 returned error can't find the container with id e71fd9f302bb030bc68e8dc7ffeccb65dff4cb5e194d804b4d22713afab0de7e
Oct 30 03:13:46.103253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:46.103222    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:46.103412 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:46.103288    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:46.103412 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:46.103315    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:46.429508 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:46.429405    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" event={"ID":"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840","Type":"ContainerStarted","Data":"572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04"}
Oct 30 03:13:46.429508 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:46.429467    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" event={"ID":"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840","Type":"ContainerStarted","Data":"e71fd9f302bb030bc68e8dc7ffeccb65dff4cb5e194d804b4d22713afab0de7e"}
Oct 30 03:13:46.471233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:46.471177    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" podStartSLOduration=1.47116245 podStartE2EDuration="1.47116245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:46.468638284 +0000 UTC m=+1971.770816659" watchObservedRunningTime="2026-10-30 03:13:46.47116245 +0000 UTC m=+1971.773340816"
Oct 30 03:13:46.766756 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:46.766661    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:47.467819 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:47.467787    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:47.768082 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:47.767995    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 03:13:47.768082 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:47.768054    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 03:13:48.106727 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:48.106697    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:48.578758 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:48.578722    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:48.767362 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:48.767328    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 03:13:48.767536 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:48.767383    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 03:13:49.390607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.390563    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"]
Oct 30 03:13:49.395255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.395231    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.413350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.413329    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-537001-openshift-default-dockercfg-5v6qv\""
Oct 30 03:13:49.461223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.461168    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"]
Oct 30 03:13:49.495903 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.495868    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-podinfo\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.496073 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.495949    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.496073 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.495999    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-data\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.496217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.496145    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-credential-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.496217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.496197    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-token\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.496288 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.496242    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbsk\" (UniqueName: \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-kube-api-access-qqbsk\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.496323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.496311    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-certs\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.496400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.496385    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-envoy\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.496491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.496479    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3692aa6a-073e-4ad9-9f06-7600188291d2-istiod-ca-cert\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.597691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.597653    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.597923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.597736    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-data\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.597923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.597784    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-credential-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.597923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.597821    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-token\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.597923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.597879    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbsk\" (UniqueName: \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-kube-api-access-qqbsk\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.598179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.597971    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-certs\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.598179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.598033    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-envoy\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.598179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.598162    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3692aa6a-073e-4ad9-9f06-7600188291d2-istiod-ca-cert\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.598363 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.598209    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-data\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.598363 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.598212    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-podinfo\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.598363 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.598245    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-credential-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.598363 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.598339    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-certs\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.598534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.598451    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-socket\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.599005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.598982    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3692aa6a-073e-4ad9-9f06-7600188291d2-istiod-ca-cert\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.601216 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.601188    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-envoy\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.601364 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.601275    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-podinfo\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.616874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.616833    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbsk\" (UniqueName: \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-kube-api-access-qqbsk\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.617480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.617454    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-token\") pod \"e2e-inf-b-537001-openshift-default-74f94b549-8ws6l\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") " pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.712758 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.712676    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:49.767035 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.767006    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 03:13:49.767180 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:49.767058    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 03:13:50.075874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:50.065124    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:50.075874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:50.071673    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"]
Oct 30 03:13:50.089175 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:13:50.086990    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3692aa6a_073e_4ad9_9f06_7600188291d2.slice/crio-e428414c68e4889a1486770fbed274c21946f883b12aeb869fe3ab3958c96a4e WatchSource:0}: Error finding container e428414c68e4889a1486770fbed274c21946f883b12aeb869fe3ab3958c96a4e: Status 404 returned error can't find the container with id e428414c68e4889a1486770fbed274c21946f883b12aeb869fe3ab3958c96a4e
Oct 30 03:13:50.094131 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:50.094092    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:50.094255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:50.094170    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:50.094255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:50.094205    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:50.480906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:50.480798    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" event={"ID":"3692aa6a-073e-4ad9-9f06-7600188291d2","Type":"ContainerStarted","Data":"734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8"}
Oct 30 03:13:50.480906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:50.480865    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" event={"ID":"3692aa6a-073e-4ad9-9f06-7600188291d2","Type":"ContainerStarted","Data":"e428414c68e4889a1486770fbed274c21946f883b12aeb869fe3ab3958c96a4e"}
Oct 30 03:13:50.545872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:50.545167    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" podStartSLOduration=1.545145352 podStartE2EDuration="1.545145352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:50.518388119 +0000 UTC m=+1975.820566484" watchObservedRunningTime="2026-10-30 03:13:50.545145352 +0000 UTC m=+1975.847323718"
Oct 30 03:13:50.713113 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:50.713080    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:50.767503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:50.767423    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 03:13:50.767503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:50.767475    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 03:13:51.513936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:51.513480    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:51.669453 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:51.669419    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:51.724870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:51.718494    5061 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 03:13:51.724870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:51.718551    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 03:13:51.769956 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:51.767529    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 03:13:51.769956 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:51.767590    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 03:13:52.366872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:52.365161    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:52.483202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:52.483161    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:52.596428 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:52.596390    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:52.713560 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:52.713479    5061 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 03:13:52.713560 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:52.713528    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 03:13:52.766867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:52.766825    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 03:13:52.767047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:52.766892    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 03:13:52.996691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:52.996598    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:53.713143 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:53.713108    5061 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 03:13:53.713530 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:53.713169    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 03:13:53.772926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:53.772895    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:53.773378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:53.773365    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:53.774189 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:53.774166    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:13:54.287055 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:54.287016    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:54.713943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:54.713903    5061 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 03:13:54.714328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:54.713977    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 03:13:55.466772 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:13:55.466717    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:13:55.713309 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:55.713262    5061 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 03:13:55.713490 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:55.713338    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 03:13:56.079150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:56.079102    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:56.713494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:56.713462    5061 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 03:13:56.713663 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:56.713521    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 03:13:57.682799 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:57.682768    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:57.736430 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:57.736396    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:57.737912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:57.737885    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:57.739126 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:57.739101    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:13:59.075773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:59.075740    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:59.191507 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:59.191472    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:59.371288 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:59.371207    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:59.534835 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:13:59.534796    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:00.226601 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:00.226569    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:00.761184 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:00.761144    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:00.897899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:00.895631    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:01.324889 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:01.324840    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:02.938937 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:02.938901    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:03.063903 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:03.057220    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:03.383513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:03.383475    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:04.424069 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:04.424034    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:04.555360 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:04.555322    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:05.509639 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:05.509599    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:06.180027 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:06.179971    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:06.321786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:06.321752    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:07.098404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:07.098371    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:08.746052 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:08.746007    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:10.427479 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:10.427447    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:12.011155 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:12.011106    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:13.432519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:13.432474    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:14.769721 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:14.769684    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:15.351094 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:15.351060    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:16.970260 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:16.970226    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:18.444739 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:18.444706    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:20.068014 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:20.067977    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:22.376774 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:22.376727    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:22.582504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:22.582449    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:23.409282 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:23.409231    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:14:23.856022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:23.855990    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:24.678005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:24.677974    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:25.447606 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:14:25.447571    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:14:26.421130 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.421090    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"]
Oct 30 03:14:26.427574 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.427550    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.443698 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.443669    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"]
Oct 30 03:14:26.447658 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.447634    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:26.451131 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.451098    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d3552\"/\"openshift-service-ca.crt\""
Oct 30 03:14:26.451246 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.451210    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d3552\"/\"auth-test-model-9d3552-kserve-self-signed-certs\""
Oct 30 03:14:26.455325 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.455299    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d3552\"/\"default-dockercfg-7fmhj\""
Oct 30 03:14:26.455425 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.455337    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-9d3552\"/\"kube-root-ca.crt\""
Oct 30 03:14:26.503695 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.503664    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-model-cache\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.503887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.503801    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tmp-dir\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.503887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.503830    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tls-certs\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.504023 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.503923    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-dshm\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.504065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.504034    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-home\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.504198 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.504180    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmws9\" (UniqueName: \"kubernetes.io/projected/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-kube-api-access-mmws9\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.570658 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.570620    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:26.605202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.605176    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmws9\" (UniqueName: \"kubernetes.io/projected/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-kube-api-access-mmws9\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.605323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.605269    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-model-cache\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.605374 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.605328    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tmp-dir\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.605374 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.605345    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tls-certs\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.605374 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.605363    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-dshm\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.605533 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.605415    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-home\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.605735 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.605712    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-model-cache\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.605802 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.605728    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tmp-dir\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.605840 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.605797    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-home\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.607581 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.607564    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-dshm\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.607917 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.607899    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tls-certs\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.622665 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.622635    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmws9\" (UniqueName: \"kubernetes.io/projected/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-kube-api-access-mmws9\") pod \"auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") " pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:26.753276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:26.753173    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:27.143080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:27.142910    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"]
Oct 30 03:14:27.146023 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:14:27.145986    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b0e504f_bc39_4ede_ba1d_0aece53d1f67.slice/crio-3382a2d318ddf2a1495a412163be19c0d8d92cccce2fd6ad09f99aec167e0c5b WatchSource:0}: Error finding container 3382a2d318ddf2a1495a412163be19c0d8d92cccce2fd6ad09f99aec167e0c5b: Status 404 returned error can't find the container with id 3382a2d318ddf2a1495a412163be19c0d8d92cccce2fd6ad09f99aec167e0c5b
Oct 30 03:14:27.959814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:27.959781    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:28.051143 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:28.051093    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4" event={"ID":"5b0e504f-bc39-4ede-ba1d-0aece53d1f67","Type":"ContainerStarted","Data":"78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678"}
Oct 30 03:14:28.051143 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:28.051148    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4" event={"ID":"5b0e504f-bc39-4ede-ba1d-0aece53d1f67","Type":"ContainerStarted","Data":"3382a2d318ddf2a1495a412163be19c0d8d92cccce2fd6ad09f99aec167e0c5b"}
Oct 30 03:14:28.089547 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:28.089493    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4" podStartSLOduration=2.089477397 podStartE2EDuration="2.089477397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:28.085869873 +0000 UTC m=+2013.388048238" watchObservedRunningTime="2026-10-30 03:14:28.089477397 +0000 UTC m=+2013.391655762"
Oct 30 03:14:28.754428 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:28.754400    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:29.564471 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:29.564440    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:31.373490 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:31.373452    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:32.876150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:32.876115    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:34.563504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:34.563470    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:36.332927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:36.332804    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:36.754643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:36.754555    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:36.767553 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:36.767521    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:37.135062 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:37.135031    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:14:38.002790 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:38.002749    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:38.971798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:38.971748    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:40.015748 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:40.015715    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:41.780513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:41.780478    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:43.272022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:43.271980    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:44.743312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:44.743273    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:44.863011 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:44.862979    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:44.977056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:44.977013    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:46.670811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:46.670775    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:48.241698 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:48.241658    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:48.682817 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:48.682780    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:49.860460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:49.860426    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:51.473052 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:51.473018    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:51.708964 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:51.708929    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:53.209077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:53.209042    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:55.264345 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:55.264233    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:55.449643 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:14:55.449607    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:14:57.309270 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:57.309234    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:58.994748 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:14:58.994713    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:00.188058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.187975    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:00.195020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.194932    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"]
Oct 30 03:15:00.205374 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.205299    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"]
Oct 30 03:15:00.206298 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.206224    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"
Oct 30 03:15:00.217896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.217823    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"
Oct 30 03:15:00.252101 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.252028    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"]
Oct 30 03:15:00.287219 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.284999    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"]
Oct 30 03:15:00.298634 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.298238    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:15:00.310655 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.310029    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"]
Oct 30 03:15:00.322211 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.322149    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"]
Oct 30 03:15:00.322478 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.322317    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"
Oct 30 03:15:00.344931 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.330213    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"]
Oct 30 03:15:00.384945 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.383579    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093cf11c-da72-45fa-9cd9-0bff00689ee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5\" (UID: \"093cf11c-da72-45fa-9cd9-0bff00689ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"
Oct 30 03:15:00.384945 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.383750    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwwng\" (UniqueName: \"kubernetes.io/projected/b4662c0e-a503-4a93-b4e3-25b672eb45a7-kube-api-access-qwwng\") pod \"maas-api-key-cleanup-29888835-hcst2\" (UID: \"b4662c0e-a503-4a93-b4e3-25b672eb45a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"
Oct 30 03:15:00.384945 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.383811    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4662c0e-a503-4a93-b4e3-25b672eb45a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888835-hcst2\" (UID: \"b4662c0e-a503-4a93-b4e3-25b672eb45a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"
Oct 30 03:15:00.394875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.394064    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"]
Oct 30 03:15:00.404181 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.396952    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"
Oct 30 03:15:00.404181 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.398607    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mhj7\" (UniqueName: \"kubernetes.io/projected/093cf11c-da72-45fa-9cd9-0bff00689ee2-kube-api-access-9mhj7\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5\" (UID: \"093cf11c-da72-45fa-9cd9-0bff00689ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"
Oct 30 03:15:00.482924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.473522    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"
Oct 30 03:15:00.512976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.512935    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz\" (UID: \"617c78a8-b288-4ad7-ab44-c8fcbf4ff339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"
Oct 30 03:15:00.513156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.513036    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-969qw\" (UniqueName: \"kubernetes.io/projected/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-kube-api-access-969qw\") pod \"maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz\" (UID: \"617c78a8-b288-4ad7-ab44-c8fcbf4ff339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"
Oct 30 03:15:00.513156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.513121    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb54g\" (UniqueName: \"kubernetes.io/projected/1e2c478d-0293-463d-b338-6da71782c4dd-kube-api-access-xb54g\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc\" (UID: \"1e2c478d-0293-463d-b338-6da71782c4dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"
Oct 30 03:15:00.513275 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.513155    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mhj7\" (UniqueName: \"kubernetes.io/projected/093cf11c-da72-45fa-9cd9-0bff00689ee2-kube-api-access-9mhj7\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5\" (UID: \"093cf11c-da72-45fa-9cd9-0bff00689ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"
Oct 30 03:15:00.513275 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.513259    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2c478d-0293-463d-b338-6da71782c4dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc\" (UID: \"1e2c478d-0293-463d-b338-6da71782c4dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"
Oct 30 03:15:00.513383 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.513305    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093cf11c-da72-45fa-9cd9-0bff00689ee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5\" (UID: \"093cf11c-da72-45fa-9cd9-0bff00689ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"
Oct 30 03:15:00.513456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.513437    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwwng\" (UniqueName: \"kubernetes.io/projected/b4662c0e-a503-4a93-b4e3-25b672eb45a7-kube-api-access-qwwng\") pod \"maas-api-key-cleanup-29888835-hcst2\" (UID: \"b4662c0e-a503-4a93-b4e3-25b672eb45a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"
Oct 30 03:15:00.522868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.514019    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csjks\" (UniqueName: \"kubernetes.io/projected/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-kube-api-access-csjks\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c\" (UID: \"ab1b496f-4d9d-418d-b7c5-1868f9c8349e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"
Oct 30 03:15:00.522868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.514082    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4662c0e-a503-4a93-b4e3-25b672eb45a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888835-hcst2\" (UID: \"b4662c0e-a503-4a93-b4e3-25b672eb45a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"
Oct 30 03:15:00.522868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.514135    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c\" (UID: \"ab1b496f-4d9d-418d-b7c5-1868f9c8349e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"
Oct 30 03:15:00.522868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.514377    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093cf11c-da72-45fa-9cd9-0bff00689ee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5\" (UID: \"093cf11c-da72-45fa-9cd9-0bff00689ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"
Oct 30 03:15:00.522868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.515035    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4662c0e-a503-4a93-b4e3-25b672eb45a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888835-hcst2\" (UID: \"b4662c0e-a503-4a93-b4e3-25b672eb45a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"
Oct 30 03:15:00.558629 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.558592    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"]
Oct 30 03:15:00.567445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.567407    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mhj7\" (UniqueName: \"kubernetes.io/projected/093cf11c-da72-45fa-9cd9-0bff00689ee2-kube-api-access-9mhj7\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5\" (UID: \"093cf11c-da72-45fa-9cd9-0bff00689ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"
Oct 30 03:15:00.572114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.572088    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"
Oct 30 03:15:00.572281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.572264    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"]
Oct 30 03:15:00.576536 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.576504    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwwng\" (UniqueName: \"kubernetes.io/projected/b4662c0e-a503-4a93-b4e3-25b672eb45a7-kube-api-access-qwwng\") pod \"maas-api-key-cleanup-29888835-hcst2\" (UID: \"b4662c0e-a503-4a93-b4e3-25b672eb45a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"
Oct 30 03:15:00.583553 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.583525    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"]
Oct 30 03:15:00.589400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.589378    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"]
Oct 30 03:15:00.589494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.589483    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"
Oct 30 03:15:00.593705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.593682    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"]
Oct 30 03:15:00.597593 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.597543    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"]
Oct 30 03:15:00.607334 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.607311    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"
Oct 30 03:15:00.616233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.616206    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csjks\" (UniqueName: \"kubernetes.io/projected/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-kube-api-access-csjks\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c\" (UID: \"ab1b496f-4d9d-418d-b7c5-1868f9c8349e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"
Oct 30 03:15:00.616348 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.616296    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c\" (UID: \"ab1b496f-4d9d-418d-b7c5-1868f9c8349e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"
Oct 30 03:15:00.616462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.616445    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz\" (UID: \"617c78a8-b288-4ad7-ab44-c8fcbf4ff339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"
Oct 30 03:15:00.616539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.616522    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-969qw\" (UniqueName: \"kubernetes.io/projected/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-kube-api-access-969qw\") pod \"maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz\" (UID: \"617c78a8-b288-4ad7-ab44-c8fcbf4ff339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"
Oct 30 03:15:00.616633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.616617    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb54g\" (UniqueName: \"kubernetes.io/projected/1e2c478d-0293-463d-b338-6da71782c4dd-kube-api-access-xb54g\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc\" (UID: \"1e2c478d-0293-463d-b338-6da71782c4dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"
Oct 30 03:15:00.616745 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.616726    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2c478d-0293-463d-b338-6da71782c4dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc\" (UID: \"1e2c478d-0293-463d-b338-6da71782c4dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"
Oct 30 03:15:00.617325 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.617238    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz\" (UID: \"617c78a8-b288-4ad7-ab44-c8fcbf4ff339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"
Oct 30 03:15:00.617325 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.617275    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c\" (UID: \"ab1b496f-4d9d-418d-b7c5-1868f9c8349e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"
Oct 30 03:15:00.617595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.617574    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2c478d-0293-463d-b338-6da71782c4dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc\" (UID: \"1e2c478d-0293-463d-b338-6da71782c4dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"
Oct 30 03:15:00.634059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.634028    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csjks\" (UniqueName: \"kubernetes.io/projected/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-kube-api-access-csjks\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c\" (UID: \"ab1b496f-4d9d-418d-b7c5-1868f9c8349e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"
Oct 30 03:15:00.634579 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.634551    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb54g\" (UniqueName: \"kubernetes.io/projected/1e2c478d-0293-463d-b338-6da71782c4dd-kube-api-access-xb54g\") pod \"maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc\" (UID: \"1e2c478d-0293-463d-b338-6da71782c4dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"
Oct 30 03:15:00.641884 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.641827    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-969qw\" (UniqueName: \"kubernetes.io/projected/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-kube-api-access-969qw\") pod \"maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz\" (UID: \"617c78a8-b288-4ad7-ab44-c8fcbf4ff339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"
Oct 30 03:15:00.718434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.718401    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr76q\" (UniqueName: \"kubernetes.io/projected/77755f65-11ff-4eea-8cd8-7f913bebcaaa-kube-api-access-cr76q\") pod \"maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5\" (UID: \"77755f65-11ff-4eea-8cd8-7f913bebcaaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"
Oct 30 03:15:00.718662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.718633    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp\" (UID: \"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"
Oct 30 03:15:00.718871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.718774    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77755f65-11ff-4eea-8cd8-7f913bebcaaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5\" (UID: \"77755f65-11ff-4eea-8cd8-7f913bebcaaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"
Oct 30 03:15:00.719143 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.719117    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mclql\" (UniqueName: \"kubernetes.io/projected/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-kube-api-access-mclql\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp\" (UID: \"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"
Oct 30 03:15:00.803902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.803867    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:00.820759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.820731    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp\" (UID: \"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"
Oct 30 03:15:00.821781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.820797    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77755f65-11ff-4eea-8cd8-7f913bebcaaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5\" (UID: \"77755f65-11ff-4eea-8cd8-7f913bebcaaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"
Oct 30 03:15:00.821781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.821050    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mclql\" (UniqueName: \"kubernetes.io/projected/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-kube-api-access-mclql\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp\" (UID: \"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"
Oct 30 03:15:00.821781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.821127    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr76q\" (UniqueName: \"kubernetes.io/projected/77755f65-11ff-4eea-8cd8-7f913bebcaaa-kube-api-access-cr76q\") pod \"maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5\" (UID: \"77755f65-11ff-4eea-8cd8-7f913bebcaaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"
Oct 30 03:15:00.821781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.821610    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp\" (UID: \"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"
Oct 30 03:15:00.821781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.821617    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77755f65-11ff-4eea-8cd8-7f913bebcaaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5\" (UID: \"77755f65-11ff-4eea-8cd8-7f913bebcaaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"
Oct 30 03:15:00.866398 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.864155    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mclql\" (UniqueName: \"kubernetes.io/projected/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-kube-api-access-mclql\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp\" (UID: \"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"
Oct 30 03:15:00.866398 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.864437    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"
Oct 30 03:15:00.866398 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.866142    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr76q\" (UniqueName: \"kubernetes.io/projected/77755f65-11ff-4eea-8cd8-7f913bebcaaa-kube-api-access-cr76q\") pod \"maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5\" (UID: \"77755f65-11ff-4eea-8cd8-7f913bebcaaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"
Oct 30 03:15:00.869086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.869061    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"
Oct 30 03:15:00.886382 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.886348    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"
Oct 30 03:15:00.912829 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.912786    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"
Oct 30 03:15:00.917564 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.917536    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"
Oct 30 03:15:00.927128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.927100    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"
Oct 30 03:15:00.928395 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:00.928367    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"]
Oct 30 03:15:01.212109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:01.212078    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"]
Oct 30 03:15:01.213420 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:15:01.213386    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1b496f_4d9d_418d_b7c5_1868f9c8349e.slice/crio-b420c41b82725d842db3457c833e36314d75a1858321dcf663ab1172b6998cd3 WatchSource:0}: Error finding container b420c41b82725d842db3457c833e36314d75a1858321dcf663ab1172b6998cd3: Status 404 returned error can't find the container with id b420c41b82725d842db3457c833e36314d75a1858321dcf663ab1172b6998cd3
Oct 30 03:15:01.340810 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:01.340781    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"]
Oct 30 03:15:01.363038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:01.363014    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"]
Oct 30 03:15:01.422494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:01.422450    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"]
Oct 30 03:15:01.450768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:01.450746    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"]
Oct 30 03:15:01.519087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:01.519060    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"]
Oct 30 03:15:01.524820 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:15:01.524787    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617c78a8_b288_4ad7_ab44_c8fcbf4ff339.slice/crio-a8995a8549a381b71291269ff029b08771bf7ca7e4cfb1aac72bb93bf7c494bc WatchSource:0}: Error finding container a8995a8549a381b71291269ff029b08771bf7ca7e4cfb1aac72bb93bf7c494bc: Status 404 returned error can't find the container with id a8995a8549a381b71291269ff029b08771bf7ca7e4cfb1aac72bb93bf7c494bc
Oct 30 03:15:01.542697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:01.542602    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c" event={"ID":"ab1b496f-4d9d-418d-b7c5-1868f9c8349e","Type":"ContainerStarted","Data":"b420c41b82725d842db3457c833e36314d75a1858321dcf663ab1172b6998cd3"}
Oct 30 03:15:01.545528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:01.545502    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5" event={"ID":"77755f65-11ff-4eea-8cd8-7f913bebcaaa","Type":"ContainerStarted","Data":"6b055cec967befc5844ac88331ecb23c982a9ade0b20a4598c07bc9f8491e7c1"}
Oct 30 03:15:01.547388 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:01.547354    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2" event={"ID":"b4662c0e-a503-4a93-b4e3-25b672eb45a7","Type":"ContainerStarted","Data":"209990568dc2911c7b066ddf823018990e69ffea035776d7a2847a57543adade"}
Oct 30 03:15:01.548829 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:01.548805    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp" event={"ID":"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce","Type":"ContainerStarted","Data":"03906e3c1114882adfab88ccf88c78b991cf238addc35a49294901e117c36b8e"}
Oct 30 03:15:01.550406 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:01.550378    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" event={"ID":"093cf11c-da72-45fa-9cd9-0bff00689ee2","Type":"ContainerStarted","Data":"91eee3992779de2899096b6a0453953f05f599901be892b6e395cc62d76eb421"}
Oct 30 03:15:02.561146 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.561109    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5" event={"ID":"77755f65-11ff-4eea-8cd8-7f913bebcaaa","Type":"ContainerStarted","Data":"952b844fdb404f20132901ebaa09ccc0f0479ef6b9d127f64d817b27ffa76048"}
Oct 30 03:15:02.562793 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.562768    5061 generic.go:356] "Generic (PLEG): container finished" podID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerID="39c1a653133a44b7eb726e88e5a247bee6eb3b06856252ae90279b4b0640e681" exitCode=60
Oct 30 03:15:02.562905 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.562878    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2" event={"ID":"b4662c0e-a503-4a93-b4e3-25b672eb45a7","Type":"ContainerDied","Data":"39c1a653133a44b7eb726e88e5a247bee6eb3b06856252ae90279b4b0640e681"}
Oct 30 03:15:02.563150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.563132    5061 scope.go:122] "RemoveContainer" containerID="39c1a653133a44b7eb726e88e5a247bee6eb3b06856252ae90279b4b0640e681"
Oct 30 03:15:02.564624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.564602    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz" event={"ID":"617c78a8-b288-4ad7-ab44-c8fcbf4ff339","Type":"ContainerStarted","Data":"31ec323de7bbc5728d064e3613aaf706a9b1160fb68033ecac528449aefe6f1c"}
Oct 30 03:15:02.564735 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.564643    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz" event={"ID":"617c78a8-b288-4ad7-ab44-c8fcbf4ff339","Type":"ContainerStarted","Data":"a8995a8549a381b71291269ff029b08771bf7ca7e4cfb1aac72bb93bf7c494bc"}
Oct 30 03:15:02.566383 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.566363    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp" event={"ID":"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce","Type":"ContainerStarted","Data":"4524a1e012425dd36775a0e66a4aa809fa7076a4826b40f9f602426566ccf1d5"}
Oct 30 03:15:02.568120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.568094    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" event={"ID":"093cf11c-da72-45fa-9cd9-0bff00689ee2","Type":"ContainerStarted","Data":"7573cd0adcf5d1ae3db82f8036a50fc63660ff884fc281b33bf6469e6e490a16"}
Oct 30 03:15:02.569928 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.569903    5061 generic.go:356] "Generic (PLEG): container finished" podID="1e2c478d-0293-463d-b338-6da71782c4dd" containerID="a835e85f9cd7465915cdf9efa0beda8c1d83d90395a516248c27b3aa2fb3f71d" exitCode=60
Oct 30 03:15:02.570028 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.569984    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc" event={"ID":"1e2c478d-0293-463d-b338-6da71782c4dd","Type":"ContainerDied","Data":"a835e85f9cd7465915cdf9efa0beda8c1d83d90395a516248c27b3aa2fb3f71d"}
Oct 30 03:15:02.570104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.570030    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc" event={"ID":"1e2c478d-0293-463d-b338-6da71782c4dd","Type":"ContainerStarted","Data":"455b050a7dfdd87e0166146ff58e486e5d112bde18b3d98de717352420ff11e4"}
Oct 30 03:15:02.570182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.570166    5061 scope.go:122] "RemoveContainer" containerID="a835e85f9cd7465915cdf9efa0beda8c1d83d90395a516248c27b3aa2fb3f71d"
Oct 30 03:15:02.571594 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.571569    5061 generic.go:356] "Generic (PLEG): container finished" podID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerID="4cd940f2b180c13085e32f994fff41fe16c061acd5c272ba9c235a3947fd9859" exitCode=60
Oct 30 03:15:02.571668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.571620    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c" event={"ID":"ab1b496f-4d9d-418d-b7c5-1868f9c8349e","Type":"ContainerDied","Data":"4cd940f2b180c13085e32f994fff41fe16c061acd5c272ba9c235a3947fd9859"}
Oct 30 03:15:02.571820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.571806    5061 scope.go:122] "RemoveContainer" containerID="4cd940f2b180c13085e32f994fff41fe16c061acd5c272ba9c235a3947fd9859"
Oct 30 03:15:02.604086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.604024    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5" podStartSLOduration=2.604004448 podStartE2EDuration="2.604004448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:02.596072078 +0000 UTC m=+2047.898250443" watchObservedRunningTime="2026-10-30 03:15:02.604004448 +0000 UTC m=+2047.906182814"
Oct 30 03:15:02.672562 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.672494    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz" podStartSLOduration=2.672479405 podStartE2EDuration="2.672479405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:02.667403382 +0000 UTC m=+2047.969581743" watchObservedRunningTime="2026-10-30 03:15:02.672479405 +0000 UTC m=+2047.974657770"
Oct 30 03:15:02.705681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.705622    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" podStartSLOduration=2.705608324 podStartE2EDuration="2.705608324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:02.703071013 +0000 UTC m=+2048.005249380" watchObservedRunningTime="2026-10-30 03:15:02.705608324 +0000 UTC m=+2048.007786689"
Oct 30 03:15:02.792825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.792744    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:02.833933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:02.828502    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp" podStartSLOduration=2.828480853 podStartE2EDuration="2.828480853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:02.807321967 +0000 UTC m=+2048.109500333" watchObservedRunningTime="2026-10-30 03:15:02.828480853 +0000 UTC m=+2048.130659219"
Oct 30 03:15:03.587370 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.587293    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c" event={"ID":"ab1b496f-4d9d-418d-b7c5-1868f9c8349e","Type":"ContainerStarted","Data":"57406c0c9864bcbd6204975bbb51bd412e81e02fa0eefeea9354ac6c1ae1e790"}
Oct 30 03:15:03.591818 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.591747    5061 generic.go:356] "Generic (PLEG): container finished" podID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerID="952b844fdb404f20132901ebaa09ccc0f0479ef6b9d127f64d817b27ffa76048" exitCode=60
Oct 30 03:15:03.592245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.591915    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5" event={"ID":"77755f65-11ff-4eea-8cd8-7f913bebcaaa","Type":"ContainerDied","Data":"952b844fdb404f20132901ebaa09ccc0f0479ef6b9d127f64d817b27ffa76048"}
Oct 30 03:15:03.592628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.592595    5061 scope.go:122] "RemoveContainer" containerID="952b844fdb404f20132901ebaa09ccc0f0479ef6b9d127f64d817b27ffa76048"
Oct 30 03:15:03.595552 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.595520    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2" event={"ID":"b4662c0e-a503-4a93-b4e3-25b672eb45a7","Type":"ContainerStarted","Data":"9c643ca5d7808e1f174dbbc30455c342c8fef4f33392df8f86f141f7d40c257c"}
Oct 30 03:15:03.598074 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.598013    5061 generic.go:356] "Generic (PLEG): container finished" podID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerID="31ec323de7bbc5728d064e3613aaf706a9b1160fb68033ecac528449aefe6f1c" exitCode=60
Oct 30 03:15:03.598497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.598092    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz" event={"ID":"617c78a8-b288-4ad7-ab44-c8fcbf4ff339","Type":"ContainerDied","Data":"31ec323de7bbc5728d064e3613aaf706a9b1160fb68033ecac528449aefe6f1c"}
Oct 30 03:15:03.599048 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.598890    5061 scope.go:122] "RemoveContainer" containerID="31ec323de7bbc5728d064e3613aaf706a9b1160fb68033ecac528449aefe6f1c"
Oct 30 03:15:03.602205 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.602131    5061 generic.go:356] "Generic (PLEG): container finished" podID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerID="4524a1e012425dd36775a0e66a4aa809fa7076a4826b40f9f602426566ccf1d5" exitCode=60
Oct 30 03:15:03.602545 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.602225    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp" event={"ID":"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce","Type":"ContainerDied","Data":"4524a1e012425dd36775a0e66a4aa809fa7076a4826b40f9f602426566ccf1d5"}
Oct 30 03:15:03.603290 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.603239    5061 scope.go:122] "RemoveContainer" containerID="4524a1e012425dd36775a0e66a4aa809fa7076a4826b40f9f602426566ccf1d5"
Oct 30 03:15:03.606164 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.606102    5061 generic.go:356] "Generic (PLEG): container finished" podID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerID="7573cd0adcf5d1ae3db82f8036a50fc63660ff884fc281b33bf6469e6e490a16" exitCode=60
Oct 30 03:15:03.606504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.606450    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" event={"ID":"093cf11c-da72-45fa-9cd9-0bff00689ee2","Type":"ContainerDied","Data":"7573cd0adcf5d1ae3db82f8036a50fc63660ff884fc281b33bf6469e6e490a16"}
Oct 30 03:15:03.607079 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.607049    5061 scope.go:122] "RemoveContainer" containerID="7573cd0adcf5d1ae3db82f8036a50fc63660ff884fc281b33bf6469e6e490a16"
Oct 30 03:15:03.613691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.613532    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc" event={"ID":"1e2c478d-0293-463d-b338-6da71782c4dd","Type":"ContainerStarted","Data":"633ca8ece3fc7368986e84e5d2fe61223863f511ce5e1d126739ce92306f8625"}
Oct 30 03:15:03.632062 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.631929    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c" podStartSLOduration=3.6318721370000002 podStartE2EDuration="3.631872137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.622978756 +0000 UTC m=+2048.925157114" watchObservedRunningTime="2026-10-30 03:15:03.631872137 +0000 UTC m=+2048.934050494"
Oct 30 03:15:03.665785 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.665593    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2" podStartSLOduration=3.665551749 podStartE2EDuration="3.665551749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.658485286 +0000 UTC m=+2048.960663651" watchObservedRunningTime="2026-10-30 03:15:03.665551749 +0000 UTC m=+2048.967730113"
Oct 30 03:15:03.771015 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:03.770947    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc" podStartSLOduration=3.770927373 podStartE2EDuration="3.770927373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.766282712 +0000 UTC m=+2049.068461076" watchObservedRunningTime="2026-10-30 03:15:03.770927373 +0000 UTC m=+2049.073105738"
Oct 30 03:15:04.425608 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.425567    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:04.630977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.630939    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" event={"ID":"093cf11c-da72-45fa-9cd9-0bff00689ee2","Type":"ContainerStarted","Data":"e33aec0040f000b913106ed97e54d34f2022a18d9dde956bc7469ffe1b2bd68f"}
Oct 30 03:15:04.634332 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.634301    5061 generic.go:356] "Generic (PLEG): container finished" podID="1e2c478d-0293-463d-b338-6da71782c4dd" containerID="633ca8ece3fc7368986e84e5d2fe61223863f511ce5e1d126739ce92306f8625" exitCode=60
Oct 30 03:15:04.634491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.634443    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc" event={"ID":"1e2c478d-0293-463d-b338-6da71782c4dd","Type":"ContainerDied","Data":"633ca8ece3fc7368986e84e5d2fe61223863f511ce5e1d126739ce92306f8625"}
Oct 30 03:15:04.634564 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.634513    5061 scope.go:122] "RemoveContainer" containerID="a835e85f9cd7465915cdf9efa0beda8c1d83d90395a516248c27b3aa2fb3f71d"
Oct 30 03:15:04.634992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.634966    5061 scope.go:122] "RemoveContainer" containerID="633ca8ece3fc7368986e84e5d2fe61223863f511ce5e1d126739ce92306f8625"
Oct 30 03:15:04.635580 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:04.635557    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc_odh-ai-gateway-infra(1e2c478d-0293-463d-b338-6da71782c4dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc" podUID="1e2c478d-0293-463d-b338-6da71782c4dd"
Oct 30 03:15:04.644305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.644266    5061 generic.go:356] "Generic (PLEG): container finished" podID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerID="57406c0c9864bcbd6204975bbb51bd412e81e02fa0eefeea9354ac6c1ae1e790" exitCode=60
Oct 30 03:15:04.644419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.644350    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c" event={"ID":"ab1b496f-4d9d-418d-b7c5-1868f9c8349e","Type":"ContainerDied","Data":"57406c0c9864bcbd6204975bbb51bd412e81e02fa0eefeea9354ac6c1ae1e790"}
Oct 30 03:15:04.644673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.644655    5061 scope.go:122] "RemoveContainer" containerID="57406c0c9864bcbd6204975bbb51bd412e81e02fa0eefeea9354ac6c1ae1e790"
Oct 30 03:15:04.645078 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:04.645038    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c_odh-ai-gateway-infra(ab1b496f-4d9d-418d-b7c5-1868f9c8349e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e"
Oct 30 03:15:04.646914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.646832    5061 generic.go:356] "Generic (PLEG): container finished" podID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerID="9c643ca5d7808e1f174dbbc30455c342c8fef4f33392df8f86f141f7d40c257c" exitCode=60
Oct 30 03:15:04.647006 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.646949    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2" event={"ID":"b4662c0e-a503-4a93-b4e3-25b672eb45a7","Type":"ContainerDied","Data":"9c643ca5d7808e1f174dbbc30455c342c8fef4f33392df8f86f141f7d40c257c"}
Oct 30 03:15:04.647353 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.647329    5061 scope.go:122] "RemoveContainer" containerID="9c643ca5d7808e1f174dbbc30455c342c8fef4f33392df8f86f141f7d40c257c"
Oct 30 03:15:04.647788 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:04.647764    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888835-hcst2_odh-ai-gateway-infra(b4662c0e-a503-4a93-b4e3-25b672eb45a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7"
Oct 30 03:15:04.650984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.650966    5061 scope.go:122] "RemoveContainer" containerID="4cd940f2b180c13085e32f994fff41fe16c061acd5c272ba9c235a3947fd9859"
Oct 30 03:15:04.665437 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:04.665418    5061 scope.go:122] "RemoveContainer" containerID="39c1a653133a44b7eb726e88e5a247bee6eb3b06856252ae90279b4b0640e681"
Oct 30 03:15:05.657300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.657214    5061 scope.go:122] "RemoveContainer" containerID="633ca8ece3fc7368986e84e5d2fe61223863f511ce5e1d126739ce92306f8625"
Oct 30 03:15:05.657720 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:05.657694    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc_odh-ai-gateway-infra(1e2c478d-0293-463d-b338-6da71782c4dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc" podUID="1e2c478d-0293-463d-b338-6da71782c4dd"
Oct 30 03:15:05.659121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.659088    5061 scope.go:122] "RemoveContainer" containerID="57406c0c9864bcbd6204975bbb51bd412e81e02fa0eefeea9354ac6c1ae1e790"
Oct 30 03:15:05.659508 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:05.659482    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c_odh-ai-gateway-infra(ab1b496f-4d9d-418d-b7c5-1868f9c8349e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e"
Oct 30 03:15:05.660812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.660785    5061 generic.go:356] "Generic (PLEG): container finished" podID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerID="badec3b21f9a26becc1c9633f3105f81f476f4fd92bcdd6a8ef474af4288655e" exitCode=60
Oct 30 03:15:05.660938 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.660910    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5" event={"ID":"77755f65-11ff-4eea-8cd8-7f913bebcaaa","Type":"ContainerDied","Data":"badec3b21f9a26becc1c9633f3105f81f476f4fd92bcdd6a8ef474af4288655e"}
Oct 30 03:15:05.661034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.660982    5061 scope.go:122] "RemoveContainer" containerID="952b844fdb404f20132901ebaa09ccc0f0479ef6b9d127f64d817b27ffa76048"
Oct 30 03:15:05.661240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.661220    5061 scope.go:122] "RemoveContainer" containerID="badec3b21f9a26becc1c9633f3105f81f476f4fd92bcdd6a8ef474af4288655e"
Oct 30 03:15:05.661709 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:05.661678    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5_odh-ai-gateway-infra(77755f65-11ff-4eea-8cd8-7f913bebcaaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa"
Oct 30 03:15:05.663235 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.663205    5061 scope.go:122] "RemoveContainer" containerID="9c643ca5d7808e1f174dbbc30455c342c8fef4f33392df8f86f141f7d40c257c"
Oct 30 03:15:05.663603 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:05.663576    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888835-hcst2_odh-ai-gateway-infra(b4662c0e-a503-4a93-b4e3-25b672eb45a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7"
Oct 30 03:15:05.664805 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.664784    5061 generic.go:356] "Generic (PLEG): container finished" podID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerID="e92f6587e4e3b5498d15903acd211b4fceb9107c7ee0dafe2dcb321049a3a5b8" exitCode=60
Oct 30 03:15:05.664896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.664878    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz" event={"ID":"617c78a8-b288-4ad7-ab44-c8fcbf4ff339","Type":"ContainerDied","Data":"e92f6587e4e3b5498d15903acd211b4fceb9107c7ee0dafe2dcb321049a3a5b8"}
Oct 30 03:15:05.665308 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.665283    5061 scope.go:122] "RemoveContainer" containerID="e92f6587e4e3b5498d15903acd211b4fceb9107c7ee0dafe2dcb321049a3a5b8"
Oct 30 03:15:05.665686 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:05.665662    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz_odh-ai-gateway-infra(617c78a8-b288-4ad7-ab44-c8fcbf4ff339)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339"
Oct 30 03:15:05.667096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.667070    5061 generic.go:356] "Generic (PLEG): container finished" podID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerID="a3947a8803505d93a705f3f174a8c789ac4f04d5149cac593e279fc678e55b22" exitCode=60
Oct 30 03:15:05.667181 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.667146    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp" event={"ID":"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce","Type":"ContainerDied","Data":"a3947a8803505d93a705f3f174a8c789ac4f04d5149cac593e279fc678e55b22"}
Oct 30 03:15:05.667433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.667419    5061 scope.go:122] "RemoveContainer" containerID="a3947a8803505d93a705f3f174a8c789ac4f04d5149cac593e279fc678e55b22"
Oct 30 03:15:05.667783 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:05.667758    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp_odh-ai-gateway-infra(a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce"
Oct 30 03:15:05.669133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.669109    5061 generic.go:356] "Generic (PLEG): container finished" podID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerID="e33aec0040f000b913106ed97e54d34f2022a18d9dde956bc7469ffe1b2bd68f" exitCode=60
Oct 30 03:15:05.669216 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.669190    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" event={"ID":"093cf11c-da72-45fa-9cd9-0bff00689ee2","Type":"ContainerDied","Data":"e33aec0040f000b913106ed97e54d34f2022a18d9dde956bc7469ffe1b2bd68f"}
Oct 30 03:15:05.669452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.669435    5061 scope.go:122] "RemoveContainer" containerID="e33aec0040f000b913106ed97e54d34f2022a18d9dde956bc7469ffe1b2bd68f"
Oct 30 03:15:05.669748 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:05.669731    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5_odh-ai-gateway-infra(093cf11c-da72-45fa-9cd9-0bff00689ee2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2"
Oct 30 03:15:05.676985 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.676966    5061 scope.go:122] "RemoveContainer" containerID="31ec323de7bbc5728d064e3613aaf706a9b1160fb68033ecac528449aefe6f1c"
Oct 30 03:15:05.692392 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.692350    5061 scope.go:122] "RemoveContainer" containerID="4524a1e012425dd36775a0e66a4aa809fa7076a4826b40f9f602426566ccf1d5"
Oct 30 03:15:05.708328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:05.708302    5061 scope.go:122] "RemoveContainer" containerID="7573cd0adcf5d1ae3db82f8036a50fc63660ff884fc281b33bf6469e6e490a16"
Oct 30 03:15:06.017511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:06.017415    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:06.682016 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:06.681986    5061 scope.go:122] "RemoveContainer" containerID="badec3b21f9a26becc1c9633f3105f81f476f4fd92bcdd6a8ef474af4288655e"
Oct 30 03:15:06.682558 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:06.682527    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5_odh-ai-gateway-infra(77755f65-11ff-4eea-8cd8-7f913bebcaaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa"
Oct 30 03:15:06.684220 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:06.684194    5061 scope.go:122] "RemoveContainer" containerID="e92f6587e4e3b5498d15903acd211b4fceb9107c7ee0dafe2dcb321049a3a5b8"
Oct 30 03:15:06.684701 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:06.684674    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz_odh-ai-gateway-infra(617c78a8-b288-4ad7-ab44-c8fcbf4ff339)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339"
Oct 30 03:15:06.686180 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:06.686149    5061 scope.go:122] "RemoveContainer" containerID="a3947a8803505d93a705f3f174a8c789ac4f04d5149cac593e279fc678e55b22"
Oct 30 03:15:06.686681 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:06.686643    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp_odh-ai-gateway-infra(a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce"
Oct 30 03:15:06.688164 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:06.688142    5061 scope.go:122] "RemoveContainer" containerID="e33aec0040f000b913106ed97e54d34f2022a18d9dde956bc7469ffe1b2bd68f"
Oct 30 03:15:06.688617 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:06.688592    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5_odh-ai-gateway-infra(093cf11c-da72-45fa-9cd9-0bff00689ee2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2"
Oct 30 03:15:07.846411 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:07.846378    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:09.510233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:09.510189    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:11.116399 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:11.116363    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:12.943095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:12.943029    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:14.572464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:14.572425    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:15.085582 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:15.085545    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:16.313959 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:16.313910    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:16.852641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:16.852597    5061 scope.go:122] "RemoveContainer" containerID="a3947a8803505d93a705f3f174a8c789ac4f04d5149cac593e279fc678e55b22"
Oct 30 03:15:16.852921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:16.852898    5061 scope.go:122] "RemoveContainer" containerID="badec3b21f9a26becc1c9633f3105f81f476f4fd92bcdd6a8ef474af4288655e"
Oct 30 03:15:16.853284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:16.853255    5061 scope.go:122] "RemoveContainer" containerID="57406c0c9864bcbd6204975bbb51bd412e81e02fa0eefeea9354ac6c1ae1e790"
Oct 30 03:15:16.853426 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:16.853344    5061 scope.go:122] "RemoveContainer" containerID="9c643ca5d7808e1f174dbbc30455c342c8fef4f33392df8f86f141f7d40c257c"
Oct 30 03:15:16.853592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:16.853472    5061 scope.go:122] "RemoveContainer" containerID="633ca8ece3fc7368986e84e5d2fe61223863f511ce5e1d126739ce92306f8625"
Oct 30 03:15:16.853684 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:16.853615    5061 scope.go:122] "RemoveContainer" containerID="e33aec0040f000b913106ed97e54d34f2022a18d9dde956bc7469ffe1b2bd68f"
Oct 30 03:15:17.797601 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:17.797562    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc" event={"ID":"1e2c478d-0293-463d-b338-6da71782c4dd","Type":"ContainerStarted","Data":"445876415dfac862f4191dee45751fd2e100dc4fa29481539d8fa4cc0c432632"}
Oct 30 03:15:17.800485 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:17.800436    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c" event={"ID":"ab1b496f-4d9d-418d-b7c5-1868f9c8349e","Type":"ContainerStarted","Data":"a0dc1b79c784b073ec2d8ab548c2f2d98be8123e15e9114b48b4f6b2c133daeb"}
Oct 30 03:15:17.802970 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:17.802939    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5" event={"ID":"77755f65-11ff-4eea-8cd8-7f913bebcaaa","Type":"ContainerStarted","Data":"5dfd2cf11693caa0f05909cbe854865cd1c5b867f26e5486061e350f333b3cd8"}
Oct 30 03:15:17.805300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:17.805273    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2" event={"ID":"b4662c0e-a503-4a93-b4e3-25b672eb45a7","Type":"ContainerStarted","Data":"1124e90734287219d025db05f638c4167e6591ef51ec7b1174825c22d48f4226"}
Oct 30 03:15:17.807484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:17.807454    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp" event={"ID":"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce","Type":"ContainerStarted","Data":"479ad08374c67d31cd826dc1b9e0140783cca4759db2c2d0b371c43522771f0a"}
Oct 30 03:15:17.809773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:17.809742    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" event={"ID":"093cf11c-da72-45fa-9cd9-0bff00689ee2","Type":"ContainerStarted","Data":"fd211a61f7a89e92bd4493452c3f8ec5be3aa949830e8b7b5195c20494a41a9c"}
Oct 30 03:15:18.076797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.076685    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:18.824302 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.824206    5061 generic.go:356] "Generic (PLEG): container finished" podID="1e2c478d-0293-463d-b338-6da71782c4dd" containerID="445876415dfac862f4191dee45751fd2e100dc4fa29481539d8fa4cc0c432632" exitCode=60
Oct 30 03:15:18.824793 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.824336    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc" event={"ID":"1e2c478d-0293-463d-b338-6da71782c4dd","Type":"ContainerDied","Data":"445876415dfac862f4191dee45751fd2e100dc4fa29481539d8fa4cc0c432632"}
Oct 30 03:15:18.824793 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.824398    5061 scope.go:122] "RemoveContainer" containerID="633ca8ece3fc7368986e84e5d2fe61223863f511ce5e1d126739ce92306f8625"
Oct 30 03:15:18.825125 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.825067    5061 scope.go:122] "RemoveContainer" containerID="445876415dfac862f4191dee45751fd2e100dc4fa29481539d8fa4cc0c432632"
Oct 30 03:15:18.825703 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:18.825598    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc_odh-ai-gateway-infra(1e2c478d-0293-463d-b338-6da71782c4dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc" podUID="1e2c478d-0293-463d-b338-6da71782c4dd"
Oct 30 03:15:18.830372 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.830258    5061 generic.go:356] "Generic (PLEG): container finished" podID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerID="a0dc1b79c784b073ec2d8ab548c2f2d98be8123e15e9114b48b4f6b2c133daeb" exitCode=60
Oct 30 03:15:18.830372 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.830291    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c" event={"ID":"ab1b496f-4d9d-418d-b7c5-1868f9c8349e","Type":"ContainerDied","Data":"a0dc1b79c784b073ec2d8ab548c2f2d98be8123e15e9114b48b4f6b2c133daeb"}
Oct 30 03:15:18.831046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.831023    5061 scope.go:122] "RemoveContainer" containerID="a0dc1b79c784b073ec2d8ab548c2f2d98be8123e15e9114b48b4f6b2c133daeb"
Oct 30 03:15:18.831673 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:18.831633    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c_odh-ai-gateway-infra(ab1b496f-4d9d-418d-b7c5-1868f9c8349e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e"
Oct 30 03:15:18.846253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.846208    5061 generic.go:356] "Generic (PLEG): container finished" podID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerID="5dfd2cf11693caa0f05909cbe854865cd1c5b867f26e5486061e350f333b3cd8" exitCode=60
Oct 30 03:15:18.849621 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.849579    5061 generic.go:356] "Generic (PLEG): container finished" podID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerID="1124e90734287219d025db05f638c4167e6591ef51ec7b1174825c22d48f4226" exitCode=60
Oct 30 03:15:18.851132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.851094    5061 scope.go:122] "RemoveContainer" containerID="57406c0c9864bcbd6204975bbb51bd412e81e02fa0eefeea9354ac6c1ae1e790"
Oct 30 03:15:18.858369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.858332    5061 generic.go:356] "Generic (PLEG): container finished" podID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerID="479ad08374c67d31cd826dc1b9e0140783cca4759db2c2d0b371c43522771f0a" exitCode=60
Oct 30 03:15:18.860833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.860798    5061 generic.go:356] "Generic (PLEG): container finished" podID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerID="fd211a61f7a89e92bd4493452c3f8ec5be3aa949830e8b7b5195c20494a41a9c" exitCode=60
Oct 30 03:15:18.863529 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.863445    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"]
Oct 30 03:15:18.863529 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.863523    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5" event={"ID":"77755f65-11ff-4eea-8cd8-7f913bebcaaa","Type":"ContainerDied","Data":"5dfd2cf11693caa0f05909cbe854865cd1c5b867f26e5486061e350f333b3cd8"}
Oct 30 03:15:18.863798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.863563    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2" event={"ID":"b4662c0e-a503-4a93-b4e3-25b672eb45a7","Type":"ContainerDied","Data":"1124e90734287219d025db05f638c4167e6591ef51ec7b1174825c22d48f4226"}
Oct 30 03:15:18.863798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.863591    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp" event={"ID":"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce","Type":"ContainerDied","Data":"479ad08374c67d31cd826dc1b9e0140783cca4759db2c2d0b371c43522771f0a"}
Oct 30 03:15:18.863798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.863620    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" event={"ID":"093cf11c-da72-45fa-9cd9-0bff00689ee2","Type":"ContainerDied","Data":"fd211a61f7a89e92bd4493452c3f8ec5be3aa949830e8b7b5195c20494a41a9c"}
Oct 30 03:15:18.864389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.864364    5061 scope.go:122] "RemoveContainer" containerID="5dfd2cf11693caa0f05909cbe854865cd1c5b867f26e5486061e350f333b3cd8"
Oct 30 03:15:18.864933 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:18.864903    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5_odh-ai-gateway-infra(77755f65-11ff-4eea-8cd8-7f913bebcaaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa"
Oct 30 03:15:18.865650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.865546    5061 scope.go:122] "RemoveContainer" containerID="479ad08374c67d31cd826dc1b9e0140783cca4759db2c2d0b371c43522771f0a"
Oct 30 03:15:18.865864 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.865826    5061 scope.go:122] "RemoveContainer" containerID="1124e90734287219d025db05f638c4167e6591ef51ec7b1174825c22d48f4226"
Oct 30 03:15:18.866358 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:18.866327    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888835-hcst2_odh-ai-gateway-infra(b4662c0e-a503-4a93-b4e3-25b672eb45a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7"
Oct 30 03:15:18.866442 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:18.866379    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp_odh-ai-gateway-infra(a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce"
Oct 30 03:15:18.866576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.866558    5061 scope.go:122] "RemoveContainer" containerID="fd211a61f7a89e92bd4493452c3f8ec5be3aa949830e8b7b5195c20494a41a9c"
Oct 30 03:15:18.866940 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:18.866915    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5_odh-ai-gateway-infra(093cf11c-da72-45fa-9cd9-0bff00689ee2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2"
Oct 30 03:15:18.875070 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.875014    5061 scope.go:122] "RemoveContainer" containerID="badec3b21f9a26becc1c9633f3105f81f476f4fd92bcdd6a8ef474af4288655e"
Oct 30 03:15:18.897713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.897652    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"]
Oct 30 03:15:18.912617 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.912585    5061 scope.go:122] "RemoveContainer" containerID="9c643ca5d7808e1f174dbbc30455c342c8fef4f33392df8f86f141f7d40c257c"
Oct 30 03:15:18.923044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.922996    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"]
Oct 30 03:15:18.934673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.934618    5061 scope.go:122] "RemoveContainer" containerID="a3947a8803505d93a705f3f174a8c789ac4f04d5149cac593e279fc678e55b22"
Oct 30 03:15:18.955446 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.955403    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"]
Oct 30 03:15:18.974155 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.974124    5061 scope.go:122] "RemoveContainer" containerID="e33aec0040f000b913106ed97e54d34f2022a18d9dde956bc7469ffe1b2bd68f"
Oct 30 03:15:18.996385 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:18.996317    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"]
Oct 30 03:15:19.028442 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:19.028399    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"]
Oct 30 03:15:19.705782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:19.705742    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:19.840105 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:19.840070    5061 scope.go:122] "RemoveContainer" containerID="e92f6587e4e3b5498d15903acd211b4fceb9107c7ee0dafe2dcb321049a3a5b8"
Oct 30 03:15:20.441830 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.441798    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"
Oct 30 03:15:20.448148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.448113    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"
Oct 30 03:15:20.453921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.453890    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"
Oct 30 03:15:20.458127 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.458065    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:20.459791 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.459756    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"
Oct 30 03:15:20.468327 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.468296    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"
Oct 30 03:15:20.473567 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.473538    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"
Oct 30 03:15:20.588044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.587941    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2c478d-0293-463d-b338-6da71782c4dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2c478d-0293-463d-b338-6da71782c4dd-maas-api-service-ca\") pod \"1e2c478d-0293-463d-b338-6da71782c4dd\" (UID: \"1e2c478d-0293-463d-b338-6da71782c4dd\") "
Oct 30 03:15:20.588044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.588033    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-kube-api-access-mclql\" (UniqueName: \"kubernetes.io/projected/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-kube-api-access-mclql\") pod \"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce\" (UID: \"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce\") "
Oct 30 03:15:20.588448 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.588081    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093cf11c-da72-45fa-9cd9-0bff00689ee2-kube-api-access-9mhj7\" (UniqueName: \"kubernetes.io/projected/093cf11c-da72-45fa-9cd9-0bff00689ee2-kube-api-access-9mhj7\") pod \"093cf11c-da72-45fa-9cd9-0bff00689ee2\" (UID: \"093cf11c-da72-45fa-9cd9-0bff00689ee2\") "
Oct 30 03:15:20.588448 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.588121    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4662c0e-a503-4a93-b4e3-25b672eb45a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4662c0e-a503-4a93-b4e3-25b672eb45a7-maas-api-service-ca\") pod \"b4662c0e-a503-4a93-b4e3-25b672eb45a7\" (UID: \"b4662c0e-a503-4a93-b4e3-25b672eb45a7\") "
Oct 30 03:15:20.588448 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.588149    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-kube-api-access-csjks\" (UniqueName: \"kubernetes.io/projected/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-kube-api-access-csjks\") pod \"ab1b496f-4d9d-418d-b7c5-1868f9c8349e\" (UID: \"ab1b496f-4d9d-418d-b7c5-1868f9c8349e\") "
Oct 30 03:15:20.589006 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.588450    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2c478d-0293-463d-b338-6da71782c4dd-kube-api-access-xb54g\" (UniqueName: \"kubernetes.io/projected/1e2c478d-0293-463d-b338-6da71782c4dd-kube-api-access-xb54g\") pod \"1e2c478d-0293-463d-b338-6da71782c4dd\" (UID: \"1e2c478d-0293-463d-b338-6da71782c4dd\") "
Oct 30 03:15:20.589006 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.588560    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4662c0e-a503-4a93-b4e3-25b672eb45a7-kube-api-access-qwwng\" (UniqueName: \"kubernetes.io/projected/b4662c0e-a503-4a93-b4e3-25b672eb45a7-kube-api-access-qwwng\") pod \"b4662c0e-a503-4a93-b4e3-25b672eb45a7\" (UID: \"b4662c0e-a503-4a93-b4e3-25b672eb45a7\") "
Oct 30 03:15:20.589006 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.588449    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2c478d-0293-463d-b338-6da71782c4dd-maas-api-service-ca" pod "1e2c478d-0293-463d-b338-6da71782c4dd" (UID: "1e2c478d-0293-463d-b338-6da71782c4dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:20.589006 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.588712    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-maas-api-service-ca\") pod \"ab1b496f-4d9d-418d-b7c5-1868f9c8349e\" (UID: \"ab1b496f-4d9d-418d-b7c5-1868f9c8349e\") "
Oct 30 03:15:20.589284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.588754    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-maas-api-service-ca\") pod \"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce\" (UID: \"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce\") "
Oct 30 03:15:20.589284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.589234    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77755f65-11ff-4eea-8cd8-7f913bebcaaa-kube-api-access-cr76q\" (UniqueName: \"kubernetes.io/projected/77755f65-11ff-4eea-8cd8-7f913bebcaaa-kube-api-access-cr76q\") pod \"77755f65-11ff-4eea-8cd8-7f913bebcaaa\" (UID: \"77755f65-11ff-4eea-8cd8-7f913bebcaaa\") "
Oct 30 03:15:20.589395 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.589339    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-maas-api-service-ca" pod "ab1b496f-4d9d-418d-b7c5-1868f9c8349e" (UID: "ab1b496f-4d9d-418d-b7c5-1868f9c8349e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:20.589530 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.589500    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093cf11c-da72-45fa-9cd9-0bff00689ee2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093cf11c-da72-45fa-9cd9-0bff00689ee2-maas-api-service-ca\") pod \"093cf11c-da72-45fa-9cd9-0bff00689ee2\" (UID: \"093cf11c-da72-45fa-9cd9-0bff00689ee2\") "
Oct 30 03:15:20.589615 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.589588    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77755f65-11ff-4eea-8cd8-7f913bebcaaa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77755f65-11ff-4eea-8cd8-7f913bebcaaa-maas-api-service-ca\") pod \"77755f65-11ff-4eea-8cd8-7f913bebcaaa\" (UID: \"77755f65-11ff-4eea-8cd8-7f913bebcaaa\") "
Oct 30 03:15:20.589760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.589663    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-maas-api-service-ca" pod "a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" (UID: "a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:20.590787 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.590676    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4662c0e-a503-4a93-b4e3-25b672eb45a7-maas-api-service-ca" pod "b4662c0e-a503-4a93-b4e3-25b672eb45a7" (UID: "b4662c0e-a503-4a93-b4e3-25b672eb45a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:20.590975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.590794    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2c478d-0293-463d-b338-6da71782c4dd-maas-api-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.590975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.590819    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-maas-api-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.590975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.590836    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-maas-api-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.591247 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.591220    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093cf11c-da72-45fa-9cd9-0bff00689ee2-maas-api-service-ca" pod "093cf11c-da72-45fa-9cd9-0bff00689ee2" (UID: "093cf11c-da72-45fa-9cd9-0bff00689ee2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:20.591395 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.591371    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77755f65-11ff-4eea-8cd8-7f913bebcaaa-maas-api-service-ca" pod "77755f65-11ff-4eea-8cd8-7f913bebcaaa" (UID: "77755f65-11ff-4eea-8cd8-7f913bebcaaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:20.592710 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.592686    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-kube-api-access-csjks" pod "ab1b496f-4d9d-418d-b7c5-1868f9c8349e" (UID: "ab1b496f-4d9d-418d-b7c5-1868f9c8349e"). InnerVolumeSpecName "kube-api-access-csjks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:20.592841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.592718    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4662c0e-a503-4a93-b4e3-25b672eb45a7-kube-api-access-qwwng" pod "b4662c0e-a503-4a93-b4e3-25b672eb45a7" (UID: "b4662c0e-a503-4a93-b4e3-25b672eb45a7"). InnerVolumeSpecName "kube-api-access-qwwng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:20.592841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.592735    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093cf11c-da72-45fa-9cd9-0bff00689ee2-kube-api-access-9mhj7" pod "093cf11c-da72-45fa-9cd9-0bff00689ee2" (UID: "093cf11c-da72-45fa-9cd9-0bff00689ee2"). InnerVolumeSpecName "kube-api-access-9mhj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:20.593266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.593242    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77755f65-11ff-4eea-8cd8-7f913bebcaaa-kube-api-access-cr76q" pod "77755f65-11ff-4eea-8cd8-7f913bebcaaa" (UID: "77755f65-11ff-4eea-8cd8-7f913bebcaaa"). InnerVolumeSpecName "kube-api-access-cr76q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:20.593653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.593629    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-kube-api-access-mclql" pod "a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" (UID: "a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce"). InnerVolumeSpecName "kube-api-access-mclql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:20.593700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.593673    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2c478d-0293-463d-b338-6da71782c4dd-kube-api-access-xb54g" pod "1e2c478d-0293-463d-b338-6da71782c4dd" (UID: "1e2c478d-0293-463d-b338-6da71782c4dd"). InnerVolumeSpecName "kube-api-access-xb54g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:20.648519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.648466    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:20.692104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.692066    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mclql\" (UniqueName: \"kubernetes.io/projected/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce-kube-api-access-mclql\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.692104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.692099    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mhj7\" (UniqueName: \"kubernetes.io/projected/093cf11c-da72-45fa-9cd9-0bff00689ee2-kube-api-access-9mhj7\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.692104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.692108    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4662c0e-a503-4a93-b4e3-25b672eb45a7-maas-api-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.692432 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.692117    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csjks\" (UniqueName: \"kubernetes.io/projected/ab1b496f-4d9d-418d-b7c5-1868f9c8349e-kube-api-access-csjks\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.692432 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.692127    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb54g\" (UniqueName: \"kubernetes.io/projected/1e2c478d-0293-463d-b338-6da71782c4dd-kube-api-access-xb54g\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.692432 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.692137    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwwng\" (UniqueName: \"kubernetes.io/projected/b4662c0e-a503-4a93-b4e3-25b672eb45a7-kube-api-access-qwwng\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.692432 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.692145    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr76q\" (UniqueName: \"kubernetes.io/projected/77755f65-11ff-4eea-8cd8-7f913bebcaaa-kube-api-access-cr76q\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.692432 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.692155    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093cf11c-da72-45fa-9cd9-0bff00689ee2-maas-api-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.692432 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.692164    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77755f65-11ff-4eea-8cd8-7f913bebcaaa-maas-api-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.905253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.905214    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"
Oct 30 03:15:20.905934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.905221    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc" event={"ID":"1e2c478d-0293-463d-b338-6da71782c4dd","Type":"ContainerDied","Data":"455b050a7dfdd87e0166146ff58e486e5d112bde18b3d98de717352420ff11e4"}
Oct 30 03:15:20.905934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.905580    5061 scope.go:122] "RemoveContainer" containerID="445876415dfac862f4191dee45751fd2e100dc4fa29481539d8fa4cc0c432632"
Oct 30 03:15:20.908819 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.908755    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c" event={"ID":"ab1b496f-4d9d-418d-b7c5-1868f9c8349e","Type":"ContainerDied","Data":"b420c41b82725d842db3457c833e36314d75a1858321dcf663ab1172b6998cd3"}
Oct 30 03:15:20.908993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.908958    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"
Oct 30 03:15:20.913212 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.912894    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"
Oct 30 03:15:20.913727 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.913694    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5" event={"ID":"77755f65-11ff-4eea-8cd8-7f913bebcaaa","Type":"ContainerDied","Data":"6b055cec967befc5844ac88331ecb23c982a9ade0b20a4598c07bc9f8491e7c1"}
Oct 30 03:15:20.924557 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.924515    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2" event={"ID":"b4662c0e-a503-4a93-b4e3-25b672eb45a7","Type":"ContainerDied","Data":"209990568dc2911c7b066ddf823018990e69ffea035776d7a2847a57543adade"}
Oct 30 03:15:20.924957 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.924935    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"
Oct 30 03:15:20.930099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.930069    5061 scope.go:122] "RemoveContainer" containerID="a0dc1b79c784b073ec2d8ab548c2f2d98be8123e15e9114b48b4f6b2c133daeb"
Oct 30 03:15:20.956312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.956272    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz" event={"ID":"617c78a8-b288-4ad7-ab44-c8fcbf4ff339","Type":"ContainerStarted","Data":"4056f53720eaaa6929d1fe68dcf9cd562681fe468f6c9ccdbf9fe3aec802e9b7"}
Oct 30 03:15:20.960132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.960101    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"
Oct 30 03:15:20.960344 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.960102    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp" event={"ID":"a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce","Type":"ContainerDied","Data":"03906e3c1114882adfab88ccf88c78b991cf238addc35a49294901e117c36b8e"}
Oct 30 03:15:20.963130 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.963092    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5" event={"ID":"093cf11c-da72-45fa-9cd9-0bff00689ee2","Type":"ContainerDied","Data":"91eee3992779de2899096b6a0453953f05f599901be892b6e395cc62d76eb421"}
Oct 30 03:15:20.963344 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.963236    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"
Oct 30 03:15:20.971517 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.971489    5061 scope.go:122] "RemoveContainer" containerID="5dfd2cf11693caa0f05909cbe854865cd1c5b867f26e5486061e350f333b3cd8"
Oct 30 03:15:20.984450 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.984419    5061 scope.go:122] "RemoveContainer" containerID="1124e90734287219d025db05f638c4167e6591ef51ec7b1174825c22d48f4226"
Oct 30 03:15:20.998684 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:20.998642    5061 scope.go:122] "RemoveContainer" containerID="479ad08374c67d31cd826dc1b9e0140783cca4759db2c2d0b371c43522771f0a"
Oct 30 03:15:21.013195 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.013141    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"]
Oct 30 03:15:21.017607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.017161    5061 scope.go:122] "RemoveContainer" containerID="fd211a61f7a89e92bd4493452c3f8ec5be3aa949830e8b7b5195c20494a41a9c"
Oct 30 03:15:21.022210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.022160    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-hcst2"]
Oct 30 03:15:21.097310 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.097258    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"]
Oct 30 03:15:21.106661 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.106597    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ae1dd9-29888835-t7wr5"]
Oct 30 03:15:21.143077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.142949    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"]
Oct 30 03:15:21.151798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.151727    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-723ffc-29888835-f7b8c"]
Oct 30 03:15:21.178797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.178738    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"]
Oct 30 03:15:21.188773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.188692    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7d1788-29888835-8k8hc"]
Oct 30 03:15:21.214003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.213947    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"]
Oct 30 03:15:21.220532 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.220486    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f21058-29888835-6hcd5"]
Oct 30 03:15:21.250807 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.250727    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"]
Oct 30 03:15:21.257627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.257562    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7d1a1-29888835-ks4cp"]
Oct 30 03:15:21.480175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.480073    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:21.981885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.981806    5061 generic.go:356] "Generic (PLEG): container finished" podID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerID="4056f53720eaaa6929d1fe68dcf9cd562681fe468f6c9ccdbf9fe3aec802e9b7" exitCode=60
Oct 30 03:15:21.982402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.981992    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz" event={"ID":"617c78a8-b288-4ad7-ab44-c8fcbf4ff339","Type":"ContainerDied","Data":"4056f53720eaaa6929d1fe68dcf9cd562681fe468f6c9ccdbf9fe3aec802e9b7"}
Oct 30 03:15:21.982402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.982049    5061 scope.go:122] "RemoveContainer" containerID="e92f6587e4e3b5498d15903acd211b4fceb9107c7ee0dafe2dcb321049a3a5b8"
Oct 30 03:15:21.983561 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:21.983529    5061 scope.go:122] "RemoveContainer" containerID="4056f53720eaaa6929d1fe68dcf9cd562681fe468f6c9ccdbf9fe3aec802e9b7"
Oct 30 03:15:21.984786 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:21.984270    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz_odh-ai-gateway-infra(617c78a8-b288-4ad7-ab44-c8fcbf4ff339)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339"
Oct 30 03:15:22.063488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:22.063446    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"]
Oct 30 03:15:22.852027 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:22.851986    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2" path="/var/lib/kubelet/pods/093cf11c-da72-45fa-9cd9-0bff00689ee2/volumes"
Oct 30 03:15:22.852695 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:22.852677    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2c478d-0293-463d-b338-6da71782c4dd" path="/var/lib/kubelet/pods/1e2c478d-0293-463d-b338-6da71782c4dd/volumes"
Oct 30 03:15:22.853260 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:22.853243    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" path="/var/lib/kubelet/pods/77755f65-11ff-4eea-8cd8-7f913bebcaaa/volumes"
Oct 30 03:15:22.854213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:22.854193    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" path="/var/lib/kubelet/pods/a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce/volumes"
Oct 30 03:15:22.854700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:22.854684    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" path="/var/lib/kubelet/pods/ab1b496f-4d9d-418d-b7c5-1868f9c8349e/volumes"
Oct 30 03:15:22.855172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:22.855156    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" path="/var/lib/kubelet/pods/b4662c0e-a503-4a93-b4e3-25b672eb45a7/volumes"
Oct 30 03:15:23.243264 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:23.243238    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"
Oct 30 03:15:23.287314 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:23.287252    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:23.371781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:23.369284    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-kube-api-access-969qw\" (UniqueName: \"kubernetes.io/projected/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-kube-api-access-969qw\") pod \"617c78a8-b288-4ad7-ab44-c8fcbf4ff339\" (UID: \"617c78a8-b288-4ad7-ab44-c8fcbf4ff339\") "
Oct 30 03:15:23.386326 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:23.384573    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-maas-api-service-ca\") pod \"617c78a8-b288-4ad7-ab44-c8fcbf4ff339\" (UID: \"617c78a8-b288-4ad7-ab44-c8fcbf4ff339\") "
Oct 30 03:15:23.388628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:23.387130    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-maas-api-service-ca" pod "617c78a8-b288-4ad7-ab44-c8fcbf4ff339" (UID: "617c78a8-b288-4ad7-ab44-c8fcbf4ff339"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:23.393738 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:23.393696    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-kube-api-access-969qw" pod "617c78a8-b288-4ad7-ab44-c8fcbf4ff339" (UID: "617c78a8-b288-4ad7-ab44-c8fcbf4ff339"). InnerVolumeSpecName "kube-api-access-969qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:23.491775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:23.491612    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-maas-api-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:23.491775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:23.491663    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-969qw\" (UniqueName: \"kubernetes.io/projected/617c78a8-b288-4ad7-ab44-c8fcbf4ff339-kube-api-access-969qw\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:24.021156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:24.021106    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz" event={"ID":"617c78a8-b288-4ad7-ab44-c8fcbf4ff339","Type":"ContainerDied","Data":"a8995a8549a381b71291269ff029b08771bf7ca7e4cfb1aac72bb93bf7c494bc"}
Oct 30 03:15:24.021389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:24.021241    5061 scope.go:122] "RemoveContainer" containerID="4056f53720eaaa6929d1fe68dcf9cd562681fe468f6c9ccdbf9fe3aec802e9b7"
Oct 30 03:15:24.021469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:24.021443    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"
Oct 30 03:15:24.054049 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:24.054003    5061 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617c78a8_b288_4ad7_ab44_c8fcbf4ff339.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 03:15:24.073198 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:24.073153    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"]
Oct 30 03:15:24.083087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:24.083038    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2305f4-29888835-9ktpz"]
Oct 30 03:15:24.857034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:24.856980    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" path="/var/lib/kubelet/pods/617c78a8-b288-4ad7-ab44-c8fcbf4ff339/volumes"
Oct 30 03:15:24.952807 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:24.952738    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:25.455030 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:25.454987    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:15:26.496531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:26.496480    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:27.995238 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:27.995194    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:29.401467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:29.401427    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:29.975587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:29.975542    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:31.246213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:31.246111    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:31.685552 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:31.685512    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:31.808327 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:31.808287    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:32.600866 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:32.598916    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:32.807103 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:32.807055    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:33.705359 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:33.705316    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:34.021228 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:34.021126    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:35.698735 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:35.698673    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:36.262769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:36.262728    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:36.390268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:36.390218    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:37.461312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:37.461272    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:38.294689 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.294643    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:38.389388 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.389343    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"]
Oct 30 03:15:38.390351 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390312    5061 cpu_manager.go:414] "containerMap: removing container" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerName="cleanup"
Oct 30 03:15:38.390351 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390343    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390364    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390370    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390380    5061 cpu_manager.go:414] "containerMap: removing container" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390387    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390402    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1e2c478d-0293-463d-b338-6da71782c4dd" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390407    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2c478d-0293-463d-b338-6da71782c4dd" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390412    5061 cpu_manager.go:414] "containerMap: removing container" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390417    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390425    5061 cpu_manager.go:414] "containerMap: removing container" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390429    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390442    5061 cpu_manager.go:414] "containerMap: removing container" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390446    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390464    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390469    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390475    5061 cpu_manager.go:414] "containerMap: removing container" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390479    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390491    5061 cpu_manager.go:414] "containerMap: removing container" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390496    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390514    5061 cpu_manager.go:414] "containerMap: removing container" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390520    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390529    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390534    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerName="cleanup"
Oct 30 03:15:38.390528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390541    5061 cpu_manager.go:414] "containerMap: removing container" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390546    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390554    5061 cpu_manager.go:414] "containerMap: removing container" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390558    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390565    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390570    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390577    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390582    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390588    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1e2c478d-0293-463d-b338-6da71782c4dd" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390592    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2c478d-0293-463d-b338-6da71782c4dd" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390598    5061 cpu_manager.go:414] "containerMap: removing container" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390602    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390621    5061 cpu_manager.go:414] "containerMap: removing container" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390629    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390776    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390784    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390791    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2c478d-0293-463d-b338-6da71782c4dd" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390797    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390806    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390812    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390819    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2c478d-0293-463d-b338-6da71782c4dd" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390826    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390832    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390838    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390859    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="093cf11c-da72-45fa-9cd9-0bff00689ee2" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390870    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390875    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390884    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerName="cleanup"
Oct 30 03:15:38.391265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.390891    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerName="cleanup"
Oct 30 03:15:38.397647 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.397614    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:15:38.415640 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.415600    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4cf8f5\""
Oct 30 03:15:38.420234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.420189    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"]
Oct 30 03:15:38.445517 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.445455    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"]
Oct 30 03:15:38.450110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.450066    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:38.450436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.450377    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerName="cleanup"
Oct 30 03:15:38.450436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.450407    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerName="cleanup"
Oct 30 03:15:38.450436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.450431    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1e2c478d-0293-463d-b338-6da71782c4dd" containerName="cleanup"
Oct 30 03:15:38.450643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.450439    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2c478d-0293-463d-b338-6da71782c4dd" containerName="cleanup"
Oct 30 03:15:38.450741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.450726    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1b496f-4d9d-418d-b7c5-1868f9c8349e" containerName="cleanup"
Oct 30 03:15:38.450776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.450750    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88825ee-02c1-4c5f-942b-1cd6e5d7e3ce" containerName="cleanup"
Oct 30 03:15:38.450776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.450761    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="617c78a8-b288-4ad7-ab44-c8fcbf4ff339" containerName="cleanup"
Oct 30 03:15:38.450883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.450776    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4662c0e-a503-4a93-b4e3-25b672eb45a7" containerName="cleanup"
Oct 30 03:15:38.450883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.450789    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2c478d-0293-463d-b338-6da71782c4dd" containerName="cleanup"
Oct 30 03:15:38.450883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.450799    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="77755f65-11ff-4eea-8cd8-7f913bebcaaa" containerName="cleanup"
Oct 30 03:15:38.456200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.456154    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:15:38.470311 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.470194    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4cf8f5-dockercfg-mhsct\""
Oct 30 03:15:38.473339 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.473106    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"]
Oct 30 03:15:38.554966 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.542543    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssjt\" (UniqueName: \"kubernetes.io/projected/52d98e0b-6b04-4c78-81c0-b0256d9a868a-kube-api-access-fssjt\") pod \"payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw\" (UID: \"52d98e0b-6b04-4c78-81c0-b0256d9a868a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:15:38.555463 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.555409    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4qzb\" (UniqueName: \"kubernetes.io/projected/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-kube-api-access-b4qzb\") pod \"payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4\" (UID: \"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:15:38.584883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.559123    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4\" (UID: \"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:15:38.654899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.586113    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d98e0b-6b04-4c78-81c0-b0256d9a868a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw\" (UID: \"52d98e0b-6b04-4c78-81c0-b0256d9a868a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:15:38.745875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.726173    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4\" (UID: \"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:15:38.745875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.726399    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d98e0b-6b04-4c78-81c0-b0256d9a868a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw\" (UID: \"52d98e0b-6b04-4c78-81c0-b0256d9a868a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:15:38.745875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.726746    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssjt\" (UniqueName: \"kubernetes.io/projected/52d98e0b-6b04-4c78-81c0-b0256d9a868a-kube-api-access-fssjt\") pod \"payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw\" (UID: \"52d98e0b-6b04-4c78-81c0-b0256d9a868a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:15:38.745875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.726899    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4qzb\" (UniqueName: \"kubernetes.io/projected/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-kube-api-access-b4qzb\") pod \"payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4\" (UID: \"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:15:38.745875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.728792    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4\" (UID: \"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:15:38.745875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.729656    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d98e0b-6b04-4c78-81c0-b0256d9a868a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw\" (UID: \"52d98e0b-6b04-4c78-81c0-b0256d9a868a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:15:38.850033 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.824402    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4qzb\" (UniqueName: \"kubernetes.io/projected/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-kube-api-access-b4qzb\") pod \"payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4\" (UID: \"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:15:38.850033 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:38.825903    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssjt\" (UniqueName: \"kubernetes.io/projected/52d98e0b-6b04-4c78-81c0-b0256d9a868a-kube-api-access-fssjt\") pod \"payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw\" (UID: \"52d98e0b-6b04-4c78-81c0-b0256d9a868a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:15:39.017550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:39.017477    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:15:39.078777 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:39.078699    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:15:39.208965 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:39.204393    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:39.362251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:39.362219    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"]
Oct 30 03:15:39.365588 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:15:39.365554    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ae0eb6_6e7a_4cc9_8358_d8ff2a6fbcac.slice/crio-3275877b33b30103ecec22c44ce9b17d676224d1529318cf892d9552945f958a WatchSource:0}: Error finding container 3275877b33b30103ecec22c44ce9b17d676224d1529318cf892d9552945f958a: Status 404 returned error can't find the container with id 3275877b33b30103ecec22c44ce9b17d676224d1529318cf892d9552945f958a
Oct 30 03:15:39.427568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:39.427531    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"]
Oct 30 03:15:39.433710 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:15:39.433541    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d98e0b_6b04_4c78_81c0_b0256d9a868a.slice/crio-336230331e4672e8f8d62f0e34588ca2f6d83bfe84954426bbba7db14ec5dcaa WatchSource:0}: Error finding container 336230331e4672e8f8d62f0e34588ca2f6d83bfe84954426bbba7db14ec5dcaa: Status 404 returned error can't find the container with id 336230331e4672e8f8d62f0e34588ca2f6d83bfe84954426bbba7db14ec5dcaa
Oct 30 03:15:40.361490 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:40.361421    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw" event={"ID":"52d98e0b-6b04-4c78-81c0-b0256d9a868a","Type":"ContainerStarted","Data":"bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63"}
Oct 30 03:15:40.363942 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:40.363896    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:15:40.364184 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:40.363976    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw" event={"ID":"52d98e0b-6b04-4c78-81c0-b0256d9a868a","Type":"ContainerStarted","Data":"336230331e4672e8f8d62f0e34588ca2f6d83bfe84954426bbba7db14ec5dcaa"}
Oct 30 03:15:40.367464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:40.367414    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4" event={"ID":"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac","Type":"ContainerStarted","Data":"fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab"}
Oct 30 03:15:40.367717 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:40.367500    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4" event={"ID":"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac","Type":"ContainerStarted","Data":"3275877b33b30103ecec22c44ce9b17d676224d1529318cf892d9552945f958a"}
Oct 30 03:15:40.368059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:40.368029    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:15:40.403936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:40.403859    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw" podStartSLOduration=2.403829726 podStartE2EDuration="2.403829726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:40.400124719 +0000 UTC m=+2085.702303084" watchObservedRunningTime="2026-10-30 03:15:40.403829726 +0000 UTC m=+2085.706008090"
Oct 30 03:15:40.438238 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:40.438168    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4" podStartSLOduration=2.438147844 podStartE2EDuration="2.438147844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:40.43755451 +0000 UTC m=+2085.739732878" watchObservedRunningTime="2026-10-30 03:15:40.438147844 +0000 UTC m=+2085.740326209"
Oct 30 03:15:40.982753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:40.982692    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:42.677497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:42.677454    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:44.235315 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:44.235269    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:45.898117 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:45.898065    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:47.013872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:47.001563    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"]
Oct 30 03:15:47.013872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:47.001884    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerName="istio-proxy" containerID="cri-o://0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7" gracePeriod=30
Oct 30 03:15:48.259067 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.259004    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:48.285876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.267965    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:48.454345 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.429939    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Oct 30 03:15:48.454345 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.435470    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:48.491341 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.491304    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"]
Oct 30 03:15:48.492386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.492339    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerName="istio-proxy" containerID="cri-o://7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7" gracePeriod=30
Oct 30 03:15:48.540147 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.536976    5061 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Oct 30 03:15:48.765401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.765358    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"]
Oct 30 03:15:48.766822 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.766774    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy" containerID="cri-o://9649535fc8a99ea425e711695d514375a1d4c0acdb8f57569e5699c7b44aad71" gracePeriod=30
Oct 30 03:15:48.822417 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.822360    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf" podUID="24ea5b2c-abf7-4857-86c5-8c1690ee76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:48.927812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.927772    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:48.957702 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:48.950893    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:15:49.078793 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:49.078732    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"]
Oct 30 03:15:49.087342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:49.087275    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy" containerID="cri-o://7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9" gracePeriod=30
Oct 30 03:15:49.348160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:49.348026    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"]
Oct 30 03:15:49.414043 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:49.413990    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:50.074139 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:50.074085    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:50.947572 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:50.947488    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:50.974164 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:50.974030    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:15:50.980144 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:50.974371    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:15:51.073600 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:51.073555    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:51.381888 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:51.381827    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:15:51.920604 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:51.920564    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:52.390400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.390367    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:15:52.397485 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.397437    5061 patch_prober.go:29] interesting pod/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:15:52.397701 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.397523    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:15:52.461428 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.461331    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:15:52.644664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.644621    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-envoy\") pod \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") "
Oct 30 03:15:52.644915 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.644727    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istiod-ca-cert\") pod \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") "
Oct 30 03:15:52.644915 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.644908    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-socket\") pod \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") "
Oct 30 03:15:52.645077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.644991    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-credential-socket\") pod \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") "
Oct 30 03:15:52.645134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.645116    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-kube-api-access-bcjrb\" (UniqueName: \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-kube-api-access-bcjrb\") pod \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") "
Oct 30 03:15:52.645251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.645193    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-socket" pod "6d985ff0-fd99-4f30-8e24-e9122ac36f88" (UID: "6d985ff0-fd99-4f30-8e24-e9122ac36f88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:52.645462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.645243    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-data\") pod \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") "
Oct 30 03:15:52.645573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.645276    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-credential-socket" pod "6d985ff0-fd99-4f30-8e24-e9122ac36f88" (UID: "6d985ff0-fd99-4f30-8e24-e9122ac36f88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:52.645573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.645305    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istiod-ca-cert" pod "6d985ff0-fd99-4f30-8e24-e9122ac36f88" (UID: "6d985ff0-fd99-4f30-8e24-e9122ac36f88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:52.645573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.645490    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-data" pod "6d985ff0-fd99-4f30-8e24-e9122ac36f88" (UID: "6d985ff0-fd99-4f30-8e24-e9122ac36f88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:52.645573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.645554    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-podinfo\") pod \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") "
Oct 30 03:15:52.645741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.645692    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-certs\") pod \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") "
Oct 30 03:15:52.645784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.645737    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-token\" (UniqueName: \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-token\") pod \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\" (UID: \"6d985ff0-fd99-4f30-8e24-e9122ac36f88\") "
Oct 30 03:15:52.646041 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.646013    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-certs" pod "6d985ff0-fd99-4f30-8e24-e9122ac36f88" (UID: "6d985ff0-fd99-4f30-8e24-e9122ac36f88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:52.647047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.646867    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.647047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.646901    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.647047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.646917    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.647047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.646937    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.647047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.646953    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.648026 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.647999    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-kube-api-access-bcjrb" pod "6d985ff0-fd99-4f30-8e24-e9122ac36f88" (UID: "6d985ff0-fd99-4f30-8e24-e9122ac36f88"). InnerVolumeSpecName "kube-api-access-bcjrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:52.648026 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.648015    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-podinfo" pod "6d985ff0-fd99-4f30-8e24-e9122ac36f88" (UID: "6d985ff0-fd99-4f30-8e24-e9122ac36f88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:15:52.648163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.647999    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-envoy" pod "6d985ff0-fd99-4f30-8e24-e9122ac36f88" (UID: "6d985ff0-fd99-4f30-8e24-e9122ac36f88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:52.648340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.648319    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-token" pod "6d985ff0-fd99-4f30-8e24-e9122ac36f88" (UID: "6d985ff0-fd99-4f30-8e24-e9122ac36f88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:52.749673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.749557    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.749673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.749595    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.749673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.749605    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d985ff0-fd99-4f30-8e24-e9122ac36f88-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.749673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.749614    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcjrb\" (UniqueName: \"kubernetes.io/projected/6d985ff0-fd99-4f30-8e24-e9122ac36f88-kube-api-access-bcjrb\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:52.900406 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.900339    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:15:52.900657 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:52.900436    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:15:53.017505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.017404    5061 generic.go:356] "Generic (PLEG): container finished" podID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerID="0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7" exitCode=0
Oct 30 03:15:53.017505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.017472    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"
Oct 30 03:15:53.017505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.017483    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" event={"ID":"6d985ff0-fd99-4f30-8e24-e9122ac36f88","Type":"ContainerDied","Data":"0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7"}
Oct 30 03:15:53.017786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.017547    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs" event={"ID":"6d985ff0-fd99-4f30-8e24-e9122ac36f88","Type":"ContainerDied","Data":"10e1ba17a12fde256db9808138582f2e643e4eb630dd5e1e31a11f58fb996429"}
Oct 30 03:15:53.017786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.017570    5061 scope.go:122] "RemoveContainer" containerID="0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7"
Oct 30 03:15:53.030163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.030135    5061 scope.go:122] "RemoveContainer" containerID="0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7"
Oct 30 03:15:53.030571 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:53.030549    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7\": container with ID starting with 0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7 not found: ID does not exist" containerID="0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7"
Oct 30 03:15:53.030624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.030582    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7"} err="failed to get container status \"0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7\": rpc error: code = NotFound desc = could not find container \"0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7\": container with ID starting with 0095b0ba4bf4bc4cde6ca1152ed327561250a97c9be49d81867ec5bcc934b7f7 not found: ID does not exist"
Oct 30 03:15:53.068281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.068233    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"]
Oct 30 03:15:53.086476 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.079302    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ngtcs"]
Oct 30 03:15:53.588463 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.588425    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:53.959363 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.959329    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:15:53.975833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.975790    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-podinfo\") pod \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") "
Oct 30 03:15:53.976083 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.975910    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-token\" (UniqueName: \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-token\") pod \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") "
Oct 30 03:15:53.976083 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.975979    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-kube-api-access-g7xlz\" (UniqueName: \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-kube-api-access-g7xlz\") pod \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") "
Oct 30 03:15:53.976083 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.976008    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istiod-ca-cert\") pod \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") "
Oct 30 03:15:53.976083 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.976055    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-credential-socket\") pod \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") "
Oct 30 03:15:53.976318 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.976116    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-envoy\") pod \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") "
Oct 30 03:15:53.976318 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.976258    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-socket\") pod \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") "
Oct 30 03:15:53.977475 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.976437    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-data\") pod \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") "
Oct 30 03:15:53.977475 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.976512    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-certs\") pod \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\" (UID: \"aed6baf0-c209-4365-9e4c-bbd985b90a9c\") "
Oct 30 03:15:53.977475 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.976925    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istiod-ca-cert" pod "aed6baf0-c209-4365-9e4c-bbd985b90a9c" (UID: "aed6baf0-c209-4365-9e4c-bbd985b90a9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:53.977752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.977679    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:53.979240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.979187    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-data" pod "aed6baf0-c209-4365-9e4c-bbd985b90a9c" (UID: "aed6baf0-c209-4365-9e4c-bbd985b90a9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:53.979240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.979205    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-credential-socket" pod "aed6baf0-c209-4365-9e4c-bbd985b90a9c" (UID: "aed6baf0-c209-4365-9e4c-bbd985b90a9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:53.979484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.979444    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-socket" pod "aed6baf0-c209-4365-9e4c-bbd985b90a9c" (UID: "aed6baf0-c209-4365-9e4c-bbd985b90a9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:53.979664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.979632    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-certs" pod "aed6baf0-c209-4365-9e4c-bbd985b90a9c" (UID: "aed6baf0-c209-4365-9e4c-bbd985b90a9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:53.981284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.981254    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-podinfo" pod "aed6baf0-c209-4365-9e4c-bbd985b90a9c" (UID: "aed6baf0-c209-4365-9e4c-bbd985b90a9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:15:53.982564 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.982495    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-kube-api-access-g7xlz" pod "aed6baf0-c209-4365-9e4c-bbd985b90a9c" (UID: "aed6baf0-c209-4365-9e4c-bbd985b90a9c"). InnerVolumeSpecName "kube-api-access-g7xlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:53.982836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.982785    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-token" pod "aed6baf0-c209-4365-9e4c-bbd985b90a9c" (UID: "aed6baf0-c209-4365-9e4c-bbd985b90a9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:53.983036 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:53.982956    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-envoy" pod "aed6baf0-c209-4365-9e4c-bbd985b90a9c" (UID: "aed6baf0-c209-4365-9e4c-bbd985b90a9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.032659 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.032612    5061 generic.go:356] "Generic (PLEG): container finished" podID="bbf52a76-996a-455b-8179-53780ef35d17" containerID="9649535fc8a99ea425e711695d514375a1d4c0acdb8f57569e5699c7b44aad71" exitCode=0
Oct 30 03:15:54.032906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.032678    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" event={"ID":"bbf52a76-996a-455b-8179-53780ef35d17","Type":"ContainerDied","Data":"9649535fc8a99ea425e711695d514375a1d4c0acdb8f57569e5699c7b44aad71"}
Oct 30 03:15:54.038230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.038185    5061 generic.go:356] "Generic (PLEG): container finished" podID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerID="7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7" exitCode=0
Oct 30 03:15:54.038458 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.038304    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"
Oct 30 03:15:54.038458 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.038320    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" event={"ID":"aed6baf0-c209-4365-9e4c-bbd985b90a9c","Type":"ContainerDied","Data":"7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7"}
Oct 30 03:15:54.038458 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.038395    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89" event={"ID":"aed6baf0-c209-4365-9e4c-bbd985b90a9c","Type":"ContainerDied","Data":"a977f50a1e049c0b9a4aa5652662b504d0a3ff753c9bf1b9857f687b741b81ad"}
Oct 30 03:15:54.038458 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.038427    5061 scope.go:122] "RemoveContainer" containerID="7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7"
Oct 30 03:15:54.068108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.068073    5061 scope.go:122] "RemoveContainer" containerID="7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7"
Oct 30 03:15:54.068668 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:54.068635    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7\": container with ID starting with 7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7 not found: ID does not exist" containerID="7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7"
Oct 30 03:15:54.068788 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.068680    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7"} err="failed to get container status \"7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7\": rpc error: code = NotFound desc = could not find container \"7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7\": container with ID starting with 7d2b3bb79db24c4ebc7a5c2566b115b88de2431f80945be2d54d0e502192ebb7 not found: ID does not exist"
Oct 30 03:15:54.079543 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.079494    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.079543 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.079529    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.079543 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.079542    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.079842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.079553    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.079842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.079568    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7xlz\" (UniqueName: \"kubernetes.io/projected/aed6baf0-c209-4365-9e4c-bbd985b90a9c-kube-api-access-g7xlz\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.079842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.079582    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.079842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.079593    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.079842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.079606    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed6baf0-c209-4365-9e4c-bbd985b90a9c-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.091812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.091783    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 03:15:54.129720 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.129667    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"]
Oct 30 03:15:54.144397 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.144361    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-g5l89"]
Oct 30 03:15:54.181100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.180991    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf52a76-996a-455b-8179-53780ef35d17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf52a76-996a-455b-8179-53780ef35d17-istiod-ca-cert\") pod \"bbf52a76-996a-455b-8179-53780ef35d17\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") "
Oct 30 03:15:54.181100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181088    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-data\") pod \"bbf52a76-996a-455b-8179-53780ef35d17\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") "
Oct 30 03:15:54.181381 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181209    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-socket\") pod \"bbf52a76-996a-455b-8179-53780ef35d17\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") "
Oct 30 03:15:54.181381 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181314    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbf52a76-996a-455b-8179-53780ef35d17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf52a76-996a-455b-8179-53780ef35d17-istio-podinfo\") pod \"bbf52a76-996a-455b-8179-53780ef35d17\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") "
Oct 30 03:15:54.181381 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181370    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-certs\") pod \"bbf52a76-996a-455b-8179-53780ef35d17\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") "
Oct 30 03:15:54.181544 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181424    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-istio-token\" (UniqueName: \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-istio-token\") pod \"bbf52a76-996a-455b-8179-53780ef35d17\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") "
Oct 30 03:15:54.181544 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181489    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-kube-api-access-bggf8\" (UniqueName: \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-kube-api-access-bggf8\") pod \"bbf52a76-996a-455b-8179-53780ef35d17\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") "
Oct 30 03:15:54.181544 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181491    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-socket" pod "bbf52a76-996a-455b-8179-53780ef35d17" (UID: "bbf52a76-996a-455b-8179-53780ef35d17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.181544 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181511    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-data" pod "bbf52a76-996a-455b-8179-53780ef35d17" (UID: "bbf52a76-996a-455b-8179-53780ef35d17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.181544 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181534    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-credential-socket\") pod \"bbf52a76-996a-455b-8179-53780ef35d17\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") "
Oct 30 03:15:54.181819 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181567    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf52a76-996a-455b-8179-53780ef35d17-istiod-ca-cert" pod "bbf52a76-996a-455b-8179-53780ef35d17" (UID: "bbf52a76-996a-455b-8179-53780ef35d17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:54.181819 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181640    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-envoy\") pod \"bbf52a76-996a-455b-8179-53780ef35d17\" (UID: \"bbf52a76-996a-455b-8179-53780ef35d17\") "
Oct 30 03:15:54.181819 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.181724    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-certs" pod "bbf52a76-996a-455b-8179-53780ef35d17" (UID: "bbf52a76-996a-455b-8179-53780ef35d17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.182664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.182636    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-credential-socket" pod "bbf52a76-996a-455b-8179-53780ef35d17" (UID: "bbf52a76-996a-455b-8179-53780ef35d17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.182833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.182797    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf52a76-996a-455b-8179-53780ef35d17-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.183056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.183031    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.183056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.183058    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.186454 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.183074    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.186454 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.184701    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-istio-token" pod "bbf52a76-996a-455b-8179-53780ef35d17" (UID: "bbf52a76-996a-455b-8179-53780ef35d17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.186454 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.184833    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbf52a76-996a-455b-8179-53780ef35d17-istio-podinfo" pod "bbf52a76-996a-455b-8179-53780ef35d17" (UID: "bbf52a76-996a-455b-8179-53780ef35d17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:15:54.186454 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.185112    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-kube-api-access-bggf8" pod "bbf52a76-996a-455b-8179-53780ef35d17" (UID: "bbf52a76-996a-455b-8179-53780ef35d17"). InnerVolumeSpecName "kube-api-access-bggf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.186720 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.186667    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-envoy" pod "bbf52a76-996a-455b-8179-53780ef35d17" (UID: "bbf52a76-996a-455b-8179-53780ef35d17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.284825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.284774    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf52a76-996a-455b-8179-53780ef35d17-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.284825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.284817    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.284825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.284827    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bggf8\" (UniqueName: \"kubernetes.io/projected/bbf52a76-996a-455b-8179-53780ef35d17-kube-api-access-bggf8\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.284825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.284836    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.284825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.284865    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf52a76-996a-455b-8179-53780ef35d17-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.833085 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.833043    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 03:15:54.851466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.851415    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" path="/var/lib/kubelet/pods/6d985ff0-fd99-4f30-8e24-e9122ac36f88/volumes"
Oct 30 03:15:54.852124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.852105    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" path="/var/lib/kubelet/pods/aed6baf0-c209-4365-9e4c-bbd985b90a9c/volumes"
Oct 30 03:15:54.897551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.897391    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-credential-socket\") pod \"4a2c026e-88e1-4e26-abe7-889ded07492c\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") "
Oct 30 03:15:54.897551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.897551    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-token\" (UniqueName: \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-token\") pod \"4a2c026e-88e1-4e26-abe7-889ded07492c\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") "
Oct 30 03:15:54.897830 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.897606    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2c026e-88e1-4e26-abe7-889ded07492c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2c026e-88e1-4e26-abe7-889ded07492c-istiod-ca-cert\") pod \"4a2c026e-88e1-4e26-abe7-889ded07492c\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") "
Oct 30 03:15:54.897830 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.897706    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-socket\") pod \"4a2c026e-88e1-4e26-abe7-889ded07492c\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") "
Oct 30 03:15:54.897830 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.897762    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-certs\") pod \"4a2c026e-88e1-4e26-abe7-889ded07492c\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") "
Oct 30 03:15:54.897997 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.897955    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-podinfo\") pod \"4a2c026e-88e1-4e26-abe7-889ded07492c\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") "
Oct 30 03:15:54.898050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.898022    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-envoy\") pod \"4a2c026e-88e1-4e26-abe7-889ded07492c\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") "
Oct 30 03:15:54.898050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.898031    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-credential-socket" pod "4a2c026e-88e1-4e26-abe7-889ded07492c" (UID: "4a2c026e-88e1-4e26-abe7-889ded07492c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.898162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.898095    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-data\") pod \"4a2c026e-88e1-4e26-abe7-889ded07492c\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") "
Oct 30 03:15:54.898162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.898134    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-kube-api-access-gpcdw\" (UniqueName: \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-kube-api-access-gpcdw\") pod \"4a2c026e-88e1-4e26-abe7-889ded07492c\" (UID: \"4a2c026e-88e1-4e26-abe7-889ded07492c\") "
Oct 30 03:15:54.898323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.898296    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-socket" pod "4a2c026e-88e1-4e26-abe7-889ded07492c" (UID: "4a2c026e-88e1-4e26-abe7-889ded07492c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.899252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.898736    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-data" pod "4a2c026e-88e1-4e26-abe7-889ded07492c" (UID: "4a2c026e-88e1-4e26-abe7-889ded07492c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.899252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.899159    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-certs" pod "4a2c026e-88e1-4e26-abe7-889ded07492c" (UID: "4a2c026e-88e1-4e26-abe7-889ded07492c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.899628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.899561    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.899628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.899587    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.899628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.899601    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:54.900191 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.900161    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2c026e-88e1-4e26-abe7-889ded07492c-istiod-ca-cert" pod "4a2c026e-88e1-4e26-abe7-889ded07492c" (UID: "4a2c026e-88e1-4e26-abe7-889ded07492c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:54.902758 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.901356    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-podinfo" pod "4a2c026e-88e1-4e26-abe7-889ded07492c" (UID: "4a2c026e-88e1-4e26-abe7-889ded07492c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:15:54.902758 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.902283    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-envoy" pod "4a2c026e-88e1-4e26-abe7-889ded07492c" (UID: "4a2c026e-88e1-4e26-abe7-889ded07492c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:15:54.904330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.904282    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-kube-api-access-gpcdw" pod "4a2c026e-88e1-4e26-abe7-889ded07492c" (UID: "4a2c026e-88e1-4e26-abe7-889ded07492c"). InnerVolumeSpecName "kube-api-access-gpcdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:54.905114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:54.905083    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-token" pod "4a2c026e-88e1-4e26-abe7-889ded07492c" (UID: "4a2c026e-88e1-4e26-abe7-889ded07492c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:55.001491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.001442    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.001491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.001486    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2c026e-88e1-4e26-abe7-889ded07492c-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.001491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.001497    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.001828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.001507    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.001828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.001515    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2c026e-88e1-4e26-abe7-889ded07492c-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.001828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.001525    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpcdw\" (UniqueName: \"kubernetes.io/projected/4a2c026e-88e1-4e26-abe7-889ded07492c-kube-api-access-gpcdw\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.051099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.051064    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"
Oct 30 03:15:55.051334 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.051056    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl" event={"ID":"bbf52a76-996a-455b-8179-53780ef35d17","Type":"ContainerDied","Data":"d6ca83edacb486f7aecd42b2ca06717006cb00e99e025fb7e2d32c97417e0761"}
Oct 30 03:15:55.051334 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.051256    5061 scope.go:122] "RemoveContainer" containerID="9649535fc8a99ea425e711695d514375a1d4c0acdb8f57569e5699c7b44aad71"
Oct 30 03:15:55.054327 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.054291    5061 generic.go:356] "Generic (PLEG): container finished" podID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerID="7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9" exitCode=0
Oct 30 03:15:55.054522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.054351    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" event={"ID":"4a2c026e-88e1-4e26-abe7-889ded07492c","Type":"ContainerDied","Data":"7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9"}
Oct 30 03:15:55.054522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.054425    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9" event={"ID":"4a2c026e-88e1-4e26-abe7-889ded07492c","Type":"ContainerDied","Data":"dd6dced5eef01f6751fff5f628cddcb5f8f2ed9ad106fde93ebd4c0954e2372f"}
Oct 30 03:15:55.054522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.054369    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"
Oct 30 03:15:55.066429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.066400    5061 scope.go:122] "RemoveContainer" containerID="7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9"
Oct 30 03:15:55.080775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.080736    5061 scope.go:122] "RemoveContainer" containerID="7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9"
Oct 30 03:15:55.081214 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:55.081188    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9\": container with ID starting with 7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9 not found: ID does not exist" containerID="7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9"
Oct 30 03:15:55.081331 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.081223    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9"} err="failed to get container status \"7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9\": rpc error: code = NotFound desc = could not find container \"7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9\": container with ID starting with 7f5c211efb072ab8c17c5f0cf3106ab5cfa78478ecf76ba1ed19a90fdf5de0d9 not found: ID does not exist"
Oct 30 03:15:55.105050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.104961    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"]
Oct 30 03:15:55.113726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.113654    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-66zxl"]
Oct 30 03:15:55.153012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.152811    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"]
Oct 30 03:15:55.158033 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.157981    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-zjww9"]
Oct 30 03:15:55.296125 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.296090    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:15:55.297920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.297835    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:15:55.299241 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.299207    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:15:55.301026 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.301005    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:15:55.452074 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:15:55.451972    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:15:55.594667 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:55.594628    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:56.850319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:56.850287    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" path="/var/lib/kubelet/pods/4a2c026e-88e1-4e26-abe7-889ded07492c/volumes"
Oct 30 03:15:56.850810 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:56.850794    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf52a76-996a-455b-8179-53780ef35d17" path="/var/lib/kubelet/pods/bbf52a76-996a-455b-8179-53780ef35d17/volumes"
Oct 30 03:15:57.085725 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:57.085695    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:58.587754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:15:58.587716    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:00.116396 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:00.116364    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:01.805255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:01.805219    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:03.526974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:03.526937    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:04.374135 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.374095    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4" podUID="5b0e504f-bc39-4ede-ba1d-0aece53d1f67" containerName="main" containerID="cri-o://78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678" gracePeriod=45
Oct 30 03:16:04.438039 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:16:04.438008    5061 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b0e504f_bc39_4ede_ba1d_0aece53d1f67.slice/crio-conmon-78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b0e504f_bc39_4ede_ba1d_0aece53d1f67.slice/crio-78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:16:04.657169 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.657131    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:16:04.708412 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.708379    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-dshm\") pod \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") "
Oct 30 03:16:04.708606 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.708433    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tls-certs\") pod \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") "
Oct 30 03:16:04.708606 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.708513    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-model-cache\") pod \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") "
Oct 30 03:16:04.708747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.708657    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-home\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-home\") pod \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") "
Oct 30 03:16:04.708816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.708762    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-kube-api-access-mmws9\" (UniqueName: \"kubernetes.io/projected/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-kube-api-access-mmws9\") pod \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") "
Oct 30 03:16:04.708816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.708767    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-model-cache" pod "5b0e504f-bc39-4ede-ba1d-0aece53d1f67" (UID: "5b0e504f-bc39-4ede-ba1d-0aece53d1f67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:04.708943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.708928    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tmp-dir\") pod \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\" (UID: \"5b0e504f-bc39-4ede-ba1d-0aece53d1f67\") "
Oct 30 03:16:04.708998 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.708974    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-home" pod "5b0e504f-bc39-4ede-ba1d-0aece53d1f67" (UID: "5b0e504f-bc39-4ede-ba1d-0aece53d1f67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:04.709904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.709277    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tmp-dir" pod "5b0e504f-bc39-4ede-ba1d-0aece53d1f67" (UID: "5b0e504f-bc39-4ede-ba1d-0aece53d1f67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:04.709904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.709799    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:16:04.709904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.709823    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:16:04.709904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.709841    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:16:04.711421 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.711396    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-dshm" pod "5b0e504f-bc39-4ede-ba1d-0aece53d1f67" (UID: "5b0e504f-bc39-4ede-ba1d-0aece53d1f67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:16:04.711527 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.711502    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-kube-api-access-mmws9" pod "5b0e504f-bc39-4ede-ba1d-0aece53d1f67" (UID: "5b0e504f-bc39-4ede-ba1d-0aece53d1f67"). InnerVolumeSpecName "kube-api-access-mmws9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:04.711594 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.711575    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tls-certs" pod "5b0e504f-bc39-4ede-ba1d-0aece53d1f67" (UID: "5b0e504f-bc39-4ede-ba1d-0aece53d1f67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:04.811299 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.811258    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:16:04.811299 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.811304    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:16:04.811506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:04.811322    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmws9\" (UniqueName: \"kubernetes.io/projected/5b0e504f-bc39-4ede-ba1d-0aece53d1f67-kube-api-access-mmws9\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:16:05.136210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:05.136172    5061 generic.go:356] "Generic (PLEG): container finished" podID="5b0e504f-bc39-4ede-ba1d-0aece53d1f67" containerID="78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678" exitCode=0
Oct 30 03:16:05.136386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:05.136237    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"
Oct 30 03:16:05.136386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:05.136253    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4" event={"ID":"5b0e504f-bc39-4ede-ba1d-0aece53d1f67","Type":"ContainerDied","Data":"78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678"}
Oct 30 03:16:05.136386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:05.136301    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4" event={"ID":"5b0e504f-bc39-4ede-ba1d-0aece53d1f67","Type":"ContainerDied","Data":"3382a2d318ddf2a1495a412163be19c0d8d92cccce2fd6ad09f99aec167e0c5b"}
Oct 30 03:16:05.136386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:05.136324    5061 scope.go:122] "RemoveContainer" containerID="78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678"
Oct 30 03:16:05.147005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:05.146983    5061 scope.go:122] "RemoveContainer" containerID="78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678"
Oct 30 03:16:05.147282 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:16:05.147261    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678\": container with ID starting with 78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678 not found: ID does not exist" containerID="78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678"
Oct 30 03:16:05.147385 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:05.147288    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678"} err="failed to get container status \"78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678\": rpc error: code = NotFound desc = could not find container \"78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678\": container with ID starting with 78ab7ac63d7757c345618088a8cf3bd420a740f795f9c6649109e1c94b07e678 not found: ID does not exist"
Oct 30 03:16:05.184495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:05.184459    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"]
Oct 30 03:16:05.196740 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:05.196704    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-9d3552/auth-test-model-9d3552-kserve-6cf6d849d9-8p4b4"]
Oct 30 03:16:05.391988 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:05.391834    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:06.851391 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:06.851360    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0e504f-bc39-4ede-ba1d-0aece53d1f67" path="/var/lib/kubelet/pods/5b0e504f-bc39-4ede-ba1d-0aece53d1f67/volumes"
Oct 30 03:16:07.087671 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:07.087638    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:08.581132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:08.581096    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:09.741736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.741703    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"]
Oct 30 03:16:09.742673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.742643    5061 cpu_manager.go:414] "containerMap: removing container" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerName="istio-proxy"
Oct 30 03:16:09.742673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.742670    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerName="istio-proxy"
Oct 30 03:16:09.742916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.742692    5061 cpu_manager.go:414] "containerMap: removing container" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy"
Oct 30 03:16:09.742916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.742702    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy"
Oct 30 03:16:09.742916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.742712    5061 cpu_manager.go:414] "containerMap: removing container" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerName="istio-proxy"
Oct 30 03:16:09.742916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.742720    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerName="istio-proxy"
Oct 30 03:16:09.742916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.742790    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy"
Oct 30 03:16:09.742916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.742798    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy"
Oct 30 03:16:09.742916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.742860    5061 cpu_manager.go:414] "containerMap: removing container" podUID="5b0e504f-bc39-4ede-ba1d-0aece53d1f67" containerName="main"
Oct 30 03:16:09.742916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.742868    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0e504f-bc39-4ede-ba1d-0aece53d1f67" containerName="main"
Oct 30 03:16:09.743329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.743031    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2c026e-88e1-4e26-abe7-889ded07492c" containerName="istio-proxy"
Oct 30 03:16:09.743329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.743052    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0e504f-bc39-4ede-ba1d-0aece53d1f67" containerName="main"
Oct 30 03:16:09.743329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.743062    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d985ff0-fd99-4f30-8e24-e9122ac36f88" containerName="istio-proxy"
Oct 30 03:16:09.743329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.743072    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf52a76-996a-455b-8179-53780ef35d17" containerName="istio-proxy"
Oct 30 03:16:09.743329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.743079    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed6baf0-c209-4365-9e4c-bbd985b90a9c" containerName="istio-proxy"
Oct 30 03:16:09.749005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.748984    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:09.763686 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.763657    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"]
Oct 30 03:16:09.770897 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.768817    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"openshift-service-ca.crt\""
Oct 30 03:16:09.770897 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.769091    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"kube-root-ca.crt\""
Oct 30 03:16:09.772979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.772954    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"default-dockercfg-jsdxd\""
Oct 30 03:16:09.772979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.772966    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-360818\"/\"sub-test-model-360818-kserve-self-signed-certs\""
Oct 30 03:16:09.820013 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.819977    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:09.919655 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.919618    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpkpj\" (UniqueName: \"kubernetes.io/projected/cc507dc4-2b61-4056-8ae6-0a617726004c-kube-api-access-cpkpj\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:09.919813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.919785    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc507dc4-2b61-4056-8ae6-0a617726004c-tls-certs\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:09.919919 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.919896    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-tmp-dir\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:09.920051 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.920031    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-model-cache\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:09.920152 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.920136    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-home\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:09.920267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:09.920241    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-dshm\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.022704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.022612    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-dshm\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.023468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.023441    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpkpj\" (UniqueName: \"kubernetes.io/projected/cc507dc4-2b61-4056-8ae6-0a617726004c-kube-api-access-cpkpj\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.024042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.024016    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc507dc4-2b61-4056-8ae6-0a617726004c-tls-certs\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.024365 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.024346    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-tmp-dir\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.024542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.024527    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-model-cache\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.027736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.024694    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-home\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.028753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.028193    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-home\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.028948 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.028911    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-tmp-dir\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.031970 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.031934    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-model-cache\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.033082 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.032940    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc507dc4-2b61-4056-8ae6-0a617726004c-tls-certs\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.036390 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.036362    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-dshm\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.041199 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.041168    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpkpj\" (UniqueName: \"kubernetes.io/projected/cc507dc4-2b61-4056-8ae6-0a617726004c-kube-api-access-cpkpj\") pod \"sub-test-model-360818-kserve-54dd7c8cfc-xtzng\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") " pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.066786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.066755    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:10.397963 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.397930    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"]
Oct 30 03:16:10.418000 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:16:10.402878    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc507dc4_2b61_4056_8ae6_0a617726004c.slice/crio-07ba4b829e4d901d4818c1d485c7512419f4af66a7b92a08de4d5346a09b900c WatchSource:0}: Error finding container 07ba4b829e4d901d4818c1d485c7512419f4af66a7b92a08de4d5346a09b900c: Status 404 returned error can't find the container with id 07ba4b829e4d901d4818c1d485c7512419f4af66a7b92a08de4d5346a09b900c
Oct 30 03:16:10.425265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:10.425234    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:11.217914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:11.217876    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng" event={"ID":"cc507dc4-2b61-4056-8ae6-0a617726004c","Type":"ContainerStarted","Data":"b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac"}
Oct 30 03:16:11.218327 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:11.217923    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng" event={"ID":"cc507dc4-2b61-4056-8ae6-0a617726004c","Type":"ContainerStarted","Data":"07ba4b829e4d901d4818c1d485c7512419f4af66a7b92a08de4d5346a09b900c"}
Oct 30 03:16:11.256187 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:11.256122    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng" podStartSLOduration=2.256091616 podStartE2EDuration="2.256091616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:11.253723667 +0000 UTC m=+2116.555902032" watchObservedRunningTime="2026-10-30 03:16:11.256091616 +0000 UTC m=+2116.558269981"
Oct 30 03:16:12.012136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:12.012106    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:12.069224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:12.068969    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:13.678869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:13.678803    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:15.450306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:15.450269    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:16.908925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:16.908877    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:18.486747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:18.486710    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:20.079666 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:20.079057    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:20.117026 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:20.113840    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:20.309268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:20.309235    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:16:20.593609 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:20.593574    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:20.716271 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:20.716238    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:20.824758 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:20.824685    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:22.380451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:22.380416    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:23.783338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:23.783305    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:24.003086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:24.003048    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:25.447464 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:16:25.447429    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:16:25.692144 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:25.692110    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:27.170209 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:27.170170    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:28.728702 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:28.728662    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:30.271316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:30.271258    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:31.899122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:31.899058    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:32.103528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:32.103479    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:33.566738 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:33.566700    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:35.335125 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:35.335077    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:36.778420 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:36.778336    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:38.363046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:38.363016    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:39.885894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:39.885856    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:41.481676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:41.481634    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:43.129422 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:43.129384    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:44.665924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:44.665886    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:46.175585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:46.175550    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:48.008569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:48.008522    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:49.039104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:49.039071    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:49.442699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:49.442648    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:51.102251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:51.102208    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:53.014358 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:53.014293    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:54.912973 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:54.912925    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:55.450883 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:16:55.450828    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:16:56.187172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:56.187134    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:56.455913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:56.455791    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:58.246519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:58.246486    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:59.888473 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:16:59.888439    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:01.378521 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:01.378480    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:02.975553 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:02.975520    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:04.403969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:04.403935    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:05.208079 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:05.208047    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:07.305622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:07.305581    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:08.294538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:08.294504    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:09.142277 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:09.142243    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:10.776784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:10.776736    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:12.274626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:12.274553    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:13.833800 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:13.833757    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:15.525285 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:15.525246    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:15.673611 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:15.673574    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:15.792098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:15.791999    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:17.118814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:17.118780    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:18.599045 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:18.599006    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:20.498897 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:20.498834    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:22.011211 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:22.011177    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:23.638467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:23.638431    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:25.347047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:25.347014    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:25.483418 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:17:25.483375    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:17:26.703286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:26.703217    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:27.223660 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:27.223619    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:27.457038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:27.457007    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:27.600958 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:27.600924    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:28.077756 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:28.077720    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:28.476400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:28.476311    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:28.589767 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:28.589731    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:29.386514 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:29.386479    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:30.937772 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:30.937732    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:31.060250 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:31.060215    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:32.783981 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:32.783927    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:34.375009 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:34.374960    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:34.526870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:34.525827    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:34.654304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:34.654193    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:34.830253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:34.830218    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"]
Oct 30 03:17:34.830497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:34.830468    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerName="istio-proxy" containerID="cri-o://572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04" gracePeriod=30
Oct 30 03:17:34.879179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:34.879140    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:35.832060 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:35.831981    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:36.508250 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:36.508211    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:38.098814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:38.098771    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:38.776268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:38.776220    5061 patch_prober.go:29] interesting pod/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:17:38.776434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:38.776300    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:17:39.968668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:39.968634    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:40.163273 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.163248    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:17:40.246266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.246234    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-kube-api-access-xm27c\" (UniqueName: \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-kube-api-access-xm27c\") pod \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") "
Oct 30 03:17:40.246449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.246288    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-envoy\") pod \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") "
Oct 30 03:17:40.246449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.246332    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-certs\") pod \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") "
Oct 30 03:17:40.246449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.246397    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-token\" (UniqueName: \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-token\") pod \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") "
Oct 30 03:17:40.246626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.246453    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-credential-socket\") pod \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") "
Oct 30 03:17:40.246626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.246508    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-socket\") pod \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") "
Oct 30 03:17:40.246733 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.246622    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-podinfo\") pod \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") "
Oct 30 03:17:40.246733 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.246675    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istiod-ca-cert\") pod \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") "
Oct 30 03:17:40.246859 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.246800    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-data\") pod \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\" (UID: \"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840\") "
Oct 30 03:17:40.247750 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.247565    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-credential-socket" pod "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" (UID: "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.247750 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.247719    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.248040 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.248009    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-data" pod "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" (UID: "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.248384 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.248344    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-socket" pod "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" (UID: "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.248550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.248527    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istiod-ca-cert" pod "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" (UID: "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:17:40.248818 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.248797    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-certs" pod "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" (UID: "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.249274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.249248    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-kube-api-access-xm27c" pod "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" (UID: "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840"). InnerVolumeSpecName "kube-api-access-xm27c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:40.249368 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.249279    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-envoy" pod "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" (UID: "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:40.250592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.250564    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-token" pod "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" (UID: "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:40.250941 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.250917    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-podinfo" pod "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" (UID: "f6ba2b21-c794-4a5b-8c5e-b9ac006f1840"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:17:40.285004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.284970    5061 generic.go:356] "Generic (PLEG): container finished" podID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerID="572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04" exitCode=0
Oct 30 03:17:40.285435 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.285025    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" event={"ID":"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840","Type":"ContainerDied","Data":"572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04"}
Oct 30 03:17:40.285435 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.285044    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"
Oct 30 03:17:40.285435 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.285066    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf" event={"ID":"f6ba2b21-c794-4a5b-8c5e-b9ac006f1840","Type":"ContainerDied","Data":"e71fd9f302bb030bc68e8dc7ffeccb65dff4cb5e194d804b4d22713afab0de7e"}
Oct 30 03:17:40.285435 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.285089    5061 scope.go:122] "RemoveContainer" containerID="572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04"
Oct 30 03:17:40.299773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.299752    5061 scope.go:122] "RemoveContainer" containerID="572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04"
Oct 30 03:17:40.300129 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:17:40.300105    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04\": container with ID starting with 572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04 not found: ID does not exist" containerID="572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04"
Oct 30 03:17:40.300230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.300140    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04"} err="failed to get container status \"572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04\": rpc error: code = NotFound desc = could not find container \"572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04\": container with ID starting with 572a7c6cfad1d4a409d7d66e94f5d8e0980b642d06fd9525ae69c70048b01e04 not found: ID does not exist"
Oct 30 03:17:40.334044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.334008    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"]
Oct 30 03:17:40.338306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.338269    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c1fce3-openshift-default-7f6c4d6669-ds4mf"]
Oct 30 03:17:40.349709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.349685    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.349709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.349708    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.349876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.349717    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.349876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.349726    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.349876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.349736    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.349876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.349744    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.349876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.349752    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm27c\" (UniqueName: \"kubernetes.io/projected/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-kube-api-access-xm27c\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.349876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.349763    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:17:40.860342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:40.860303    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" path="/var/lib/kubelet/pods/f6ba2b21-c794-4a5b-8c5e-b9ac006f1840/volumes"
Oct 30 03:17:41.566125 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:41.566095    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:41.900077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:41.899961    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:43.176545 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:43.176506    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:44.069086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:44.069047    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:45.975264 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:45.975152    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:46.119460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:46.119417    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:48.104715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:48.104679    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:49.151326 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:49.151293    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:49.767566 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:49.767532    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:51.332223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:51.332184    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:52.840650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:52.840612    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:53.423862 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:53.423810    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:54.680650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:54.680613    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:55.449655 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:17:55.449615    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:17:56.259106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:56.259068    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:57.812735 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:57.812689    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:59.270204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:17:59.270097    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:00.761944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:00.761902    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:02.403358 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:02.403314    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:04.008292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:04.008262    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:04.215769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:04.215734    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:04.339813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:04.339775    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:05.789660 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:05.789631    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:07.310256 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:07.310220    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:08.891701 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:08.891664    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:10.684124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:10.684092    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:11.398897 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:11.398865    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:11.525162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:11.525129    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:12.392539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:12.392507    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:13.983034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:13.982997    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:15.340886 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:15.340840    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:16.069746 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.069712    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"]
Oct 30 03:18:16.070627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.070602    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerName="istio-proxy"
Oct 30 03:18:16.070627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.070623    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerName="istio-proxy"
Oct 30 03:18:16.070833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.070818    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ba2b21-c794-4a5b-8c5e-b9ac006f1840" containerName="istio-proxy"
Oct 30 03:18:16.076744 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.076694    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.100504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.100479    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"]
Oct 30 03:18:16.103353 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.103326    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2305f4-openshift-default-dockercfg-l7spl\""
Oct 30 03:18:16.149830 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.149796    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:16.203450 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.203407    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istiod-ca-cert\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.203641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.203489    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-credential-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.203641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.203564    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-certs\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.203766 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.203682    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x52gh\" (UniqueName: \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-kube-api-access-x52gh\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.203766 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.203730    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-data\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.203891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.203766    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-envoy\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.203968 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.203947    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-token\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.204023 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.204010    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.204089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.204077    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-podinfo\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.305389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.305348    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-token\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.305595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.305442    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.305595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.305492    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-podinfo\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.305595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.305557    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istiod-ca-cert\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.305780 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.305653    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-credential-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.305780 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.305714    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-certs\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.305907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.305834    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x52gh\" (UniqueName: \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-kube-api-access-x52gh\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.305967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.305932    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-data\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.306023 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.305970    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-credential-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.306023 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.305992    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-envoy\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.306217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.306160    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-certs\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.306338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.306244    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-socket\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.306338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.306268    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-data\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.306469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.306399    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istiod-ca-cert\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.308550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.308526    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-podinfo\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.308688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.308582    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-envoy\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.312186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.312153    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:16.327768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.327713    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-token\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.328070 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.328049    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x52gh\" (UniqueName: \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-kube-api-access-x52gh\") pod \"e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") " pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.394017 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.393979    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:16.466371 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.466306    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:16.766428 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.766401    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"]
Oct 30 03:18:16.768422 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:18:16.768389    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddfe1b3_af75_46ac_a5a1_5ece8c06ede2.slice/crio-ed1a04dfbde976670604cdb7be66007f524183f3bd8575efda5f9b70dc12c2e7 WatchSource:0}: Error finding container ed1a04dfbde976670604cdb7be66007f524183f3bd8575efda5f9b70dc12c2e7: Status 404 returned error can't find the container with id ed1a04dfbde976670604cdb7be66007f524183f3bd8575efda5f9b70dc12c2e7
Oct 30 03:18:16.775827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.775782    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:18:16.775962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.775902    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:18:16.775962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.775938    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:18:16.882965 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:16.882924    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:17.634874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:17.634823    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" event={"ID":"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2","Type":"ContainerStarted","Data":"cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7"}
Oct 30 03:18:17.635484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:17.634890    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" event={"ID":"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2","Type":"ContainerStarted","Data":"ed1a04dfbde976670604cdb7be66007f524183f3bd8575efda5f9b70dc12c2e7"}
Oct 30 03:18:17.693471 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:17.690992    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" podStartSLOduration=1.690971124 podStartE2EDuration="1.690971124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:17.683232414 +0000 UTC m=+2242.985410781" watchObservedRunningTime="2026-10-30 03:18:17.690971124 +0000 UTC m=+2242.993149489"
Oct 30 03:18:18.281542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:18.281503    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:18.394737 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:18.394696    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:18.396644 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:18.396612    5061 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:18:18.396811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:18.396665    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" podUID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 03:18:19.394841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:19.394809    5061 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:18:19.395231 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:19.394896    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" podUID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 03:18:19.845510 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:19.845469    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:20.394755 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:20.394718    5061 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:18:20.394998 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:20.394775    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" podUID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 03:18:21.394931 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:21.394886    5061 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:18:21.395122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:21.394949    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" podUID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 03:18:21.441873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:21.439975    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:22.395260 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:22.395223    5061 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:18:22.395650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:22.395279    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" podUID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 03:18:23.394889 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:23.394820    5061 patch_prober.go:29] interesting pod/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 03:18:23.395103 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:23.394913    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" podUID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 03:18:23.686243 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:23.686155    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:23.914593 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:23.914551    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:24.044343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:24.044312    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:24.267669 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:24.267627    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:24.451253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:24.451160    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:24.453188 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:24.453162    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:24.460098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:24.460073    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:18:25.182126 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:25.182095    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:25.371349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:25.371307    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:25.477037 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:18:25.476924    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:18:25.677675 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:25.677643    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:25.832552 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:25.832518    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:27.193137 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:27.193100    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:28.930295 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:28.930259    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:29.651652 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:29.651616    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:29.833661 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:29.833629    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:30.455332 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:30.455291    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:31.522625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:31.522587    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:31.634484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:31.634450    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:31.768644 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:31.768610    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:32.153964 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:32.153927    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:32.309134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:32.309100    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:33.809700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:33.809667    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:35.511281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:35.511244    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:37.200599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:37.200539    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:38.235103 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:38.235073    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:38.358156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:38.358124    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:38.584330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:38.584297    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:38.812869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:38.812741    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:40.562302 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:40.562160    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:42.184460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:42.184423    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:43.802286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:43.802249    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:45.719460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:45.719424    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:46.237966 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:46.237926    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:47.416934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:47.416899    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:47.967174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:47.967118    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"]
Oct 30 03:18:48.039005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:48.038965    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:49.246523 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:49.246487    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"]
Oct 30 03:18:49.247589 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:49.247556    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerName="istio-proxy" containerID="cri-o://ea737c0c4e50cc84f31640379f6929ce6464953dd1240fdcad91fac0417caf74" gracePeriod=30
Oct 30 03:18:49.289559 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:49.289521    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:49.402116 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:49.402079    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:49.557539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:49.557512    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"]
Oct 30 03:18:49.797084 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:49.797032    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:49.913587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:49.913497    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:51.255317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:51.255282    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:52.774839 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:52.774807    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:53.214217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:53.214184    5061 patch_prober.go:29] interesting pod/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:18:53.214403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:53.214249    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:18:54.288619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.288589    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:54.466230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.465807    5061 generic.go:356] "Generic (PLEG): container finished" podID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerID="ea737c0c4e50cc84f31640379f6929ce6464953dd1240fdcad91fac0417caf74" exitCode=0
Oct 30 03:18:54.466230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.466119    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" event={"ID":"f8d65367-9c08-4b32-b9d4-002d9bc4c55c","Type":"ContainerDied","Data":"ea737c0c4e50cc84f31640379f6929ce6464953dd1240fdcad91fac0417caf74"}
Oct 30 03:18:54.558145 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.558108    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf" podUID="df2fc39f-d861-4c39-aa82-fd1317b358ae" containerName="payload-processing" containerID="cri-o://aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb" gracePeriod=25
Oct 30 03:18:54.558435 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.558381    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 03:18:54.721107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.721075    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-podinfo\") pod \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") "
Oct 30 03:18:54.721387 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.721365    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-token\" (UniqueName: \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-token\") pod \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") "
Oct 30 03:18:54.721492 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.721472    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-socket\") pod \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") "
Oct 30 03:18:54.721646 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.721623    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-certs\") pod \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") "
Oct 30 03:18:54.721837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.721672    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-credential-socket\") pod \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") "
Oct 30 03:18:54.721837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.721752    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-kube-api-access-jgrbs\" (UniqueName: \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-kube-api-access-jgrbs\") pod \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") "
Oct 30 03:18:54.721837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.721795    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istiod-ca-cert\") pod \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") "
Oct 30 03:18:54.721837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.721822    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-envoy\") pod \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") "
Oct 30 03:18:54.722215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.721811    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-socket" pod "f8d65367-9c08-4b32-b9d4-002d9bc4c55c" (UID: "f8d65367-9c08-4b32-b9d4-002d9bc4c55c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:54.722215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.721919    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-data\") pod \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\" (UID: \"f8d65367-9c08-4b32-b9d4-002d9bc4c55c\") "
Oct 30 03:18:54.722215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.722026    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-credential-socket" pod "f8d65367-9c08-4b32-b9d4-002d9bc4c55c" (UID: "f8d65367-9c08-4b32-b9d4-002d9bc4c55c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:54.722215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.722203    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-data" pod "f8d65367-9c08-4b32-b9d4-002d9bc4c55c" (UID: "f8d65367-9c08-4b32-b9d4-002d9bc4c55c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:54.722447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.722374    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istiod-ca-cert" pod "f8d65367-9c08-4b32-b9d4-002d9bc4c55c" (UID: "f8d65367-9c08-4b32-b9d4-002d9bc4c55c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:54.722814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.722792    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:18:54.722993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.722819    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:18:54.722993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.722832    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:18:54.722993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.722860    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:18:54.724269 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.724091    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-certs" pod "f8d65367-9c08-4b32-b9d4-002d9bc4c55c" (UID: "f8d65367-9c08-4b32-b9d4-002d9bc4c55c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:54.730107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.730076    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-envoy" pod "f8d65367-9c08-4b32-b9d4-002d9bc4c55c" (UID: "f8d65367-9c08-4b32-b9d4-002d9bc4c55c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:54.730279 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.730255    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-podinfo" pod "f8d65367-9c08-4b32-b9d4-002d9bc4c55c" (UID: "f8d65367-9c08-4b32-b9d4-002d9bc4c55c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:18:54.730375 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.730356    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-kube-api-access-jgrbs" pod "f8d65367-9c08-4b32-b9d4-002d9bc4c55c" (UID: "f8d65367-9c08-4b32-b9d4-002d9bc4c55c"). InnerVolumeSpecName "kube-api-access-jgrbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:54.730842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.730783    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-token" pod "f8d65367-9c08-4b32-b9d4-002d9bc4c55c" (UID: "f8d65367-9c08-4b32-b9d4-002d9bc4c55c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:54.824762 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.824735    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:18:54.824762 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.824761    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgrbs\" (UniqueName: \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-kube-api-access-jgrbs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:18:54.824977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.824771    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:18:54.824977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.824781    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:18:54.824977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:54.824800    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d65367-9c08-4b32-b9d4-002d9bc4c55c-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:18:55.448213 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:18:55.448167    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:18:55.474497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:55.474463    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm" event={"ID":"f8d65367-9c08-4b32-b9d4-002d9bc4c55c","Type":"ContainerDied","Data":"b34acf3757968948896430814444c10c6c1cc8462919f6997dce3e3ebc9ec7ea"}
Oct 30 03:18:55.474497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:55.474496    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"
Oct 30 03:18:55.474706 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:55.474526    5061 scope.go:122] "RemoveContainer" containerID="ea737c0c4e50cc84f31640379f6929ce6464953dd1240fdcad91fac0417caf74"
Oct 30 03:18:55.533784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:55.533754    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"]
Oct 30 03:18:55.538436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:55.538417    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-t78nm"]
Oct 30 03:18:55.903122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:55.903087    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:56.849573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:56.849536    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" path="/var/lib/kubelet/pods/f8d65367-9c08-4b32-b9d4-002d9bc4c55c/volumes"
Oct 30 03:18:57.452741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:57.452701    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:58.225226 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:58.225189    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf" podUID="df2fc39f-d861-4c39-aa82-fd1317b358ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:18:58.647956 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:58.647909    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:59.241919 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:18:59.241838    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:00.976828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:00.976797    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:01.707206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:01.707132    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:02.778118 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:02.778083    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:02.993409 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:02.993373    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng" podUID="cc507dc4-2b61-4056-8ae6-0a617726004c" containerName="main" containerID="cri-o://b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac" gracePeriod=45
Oct 30 03:19:03.287659 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.287627    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:19:03.402195 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.402116    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc507dc4-2b61-4056-8ae6-0a617726004c-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc507dc4-2b61-4056-8ae6-0a617726004c-tls-certs\") pod \"cc507dc4-2b61-4056-8ae6-0a617726004c\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") "
Oct 30 03:19:03.402368 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.402221    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-tmp-dir\") pod \"cc507dc4-2b61-4056-8ae6-0a617726004c\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") "
Oct 30 03:19:03.402368 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.402246    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-model-cache\") pod \"cc507dc4-2b61-4056-8ae6-0a617726004c\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") "
Oct 30 03:19:03.402368 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.402352    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-home\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-home\") pod \"cc507dc4-2b61-4056-8ae6-0a617726004c\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") "
Oct 30 03:19:03.402554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.402370    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-dshm\") pod \"cc507dc4-2b61-4056-8ae6-0a617726004c\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") "
Oct 30 03:19:03.402554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.402438    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc507dc4-2b61-4056-8ae6-0a617726004c-kube-api-access-cpkpj\" (UniqueName: \"kubernetes.io/projected/cc507dc4-2b61-4056-8ae6-0a617726004c-kube-api-access-cpkpj\") pod \"cc507dc4-2b61-4056-8ae6-0a617726004c\" (UID: \"cc507dc4-2b61-4056-8ae6-0a617726004c\") "
Oct 30 03:19:03.402554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.402521    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-tmp-dir" pod "cc507dc4-2b61-4056-8ae6-0a617726004c" (UID: "cc507dc4-2b61-4056-8ae6-0a617726004c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:03.402688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.402630    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-home" pod "cc507dc4-2b61-4056-8ae6-0a617726004c" (UID: "cc507dc4-2b61-4056-8ae6-0a617726004c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:03.402688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.402647    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-model-cache" pod "cc507dc4-2b61-4056-8ae6-0a617726004c" (UID: "cc507dc4-2b61-4056-8ae6-0a617726004c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:03.403063 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.403040    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:03.403063 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.403065    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:03.403242 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.403077    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:03.405181 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.405141    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc507dc4-2b61-4056-8ae6-0a617726004c-tls-certs" pod "cc507dc4-2b61-4056-8ae6-0a617726004c" (UID: "cc507dc4-2b61-4056-8ae6-0a617726004c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:03.405181 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.405154    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc507dc4-2b61-4056-8ae6-0a617726004c-kube-api-access-cpkpj" pod "cc507dc4-2b61-4056-8ae6-0a617726004c" (UID: "cc507dc4-2b61-4056-8ae6-0a617726004c"). InnerVolumeSpecName "kube-api-access-cpkpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:03.405346 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.405224    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-dshm" pod "cc507dc4-2b61-4056-8ae6-0a617726004c" (UID: "cc507dc4-2b61-4056-8ae6-0a617726004c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:03.504769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.504729    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc507dc4-2b61-4056-8ae6-0a617726004c-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:03.504769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.504764    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc507dc4-2b61-4056-8ae6-0a617726004c-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:03.505006 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.504777    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpkpj\" (UniqueName: \"kubernetes.io/projected/cc507dc4-2b61-4056-8ae6-0a617726004c-kube-api-access-cpkpj\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:03.557581 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.557549    5061 generic.go:356] "Generic (PLEG): container finished" podID="cc507dc4-2b61-4056-8ae6-0a617726004c" containerID="b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac" exitCode=0
Oct 30 03:19:03.557728 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.557595    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"
Oct 30 03:19:03.557728 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.557631    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng" event={"ID":"cc507dc4-2b61-4056-8ae6-0a617726004c","Type":"ContainerDied","Data":"b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac"}
Oct 30 03:19:03.557728 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.557675    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng" event={"ID":"cc507dc4-2b61-4056-8ae6-0a617726004c","Type":"ContainerDied","Data":"07ba4b829e4d901d4818c1d485c7512419f4af66a7b92a08de4d5346a09b900c"}
Oct 30 03:19:03.557728 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.557691    5061 scope.go:122] "RemoveContainer" containerID="b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac"
Oct 30 03:19:03.569699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.569682    5061 scope.go:122] "RemoveContainer" containerID="b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac"
Oct 30 03:19:03.570048 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:19:03.570030    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac\": container with ID starting with b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac not found: ID does not exist" containerID="b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac"
Oct 30 03:19:03.570151 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.570052    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac"} err="failed to get container status \"b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac\": rpc error: code = NotFound desc = could not find container \"b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac\": container with ID starting with b5f4f530e425a1e769b50987860f42aef297c1994de969d4866d9ed43c1a1bac not found: ID does not exist"
Oct 30 03:19:03.604721 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.604698    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"]
Oct 30 03:19:03.614571 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:03.614540    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-360818/sub-test-model-360818-kserve-54dd7c8cfc-xtzng"]
Oct 30 03:19:04.441177 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:04.441144    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:04.849653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:04.849620    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc507dc4-2b61-4056-8ae6-0a617726004c" path="/var/lib/kubelet/pods/cc507dc4-2b61-4056-8ae6-0a617726004c/volumes"
Oct 30 03:19:05.896546 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:05.896507    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:07.530602 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:07.530565    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:08.224630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:08.224594    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf" podUID="df2fc39f-d861-4c39-aa82-fd1317b358ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:19:09.028180 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:09.028147    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:09.142305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:09.142274    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:09.258205 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:09.258170    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:09.517971 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:09.517941    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"]
Oct 30 03:19:09.550204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:09.550173    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"]
Oct 30 03:19:09.677920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:09.677839    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:09.794378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:09.793770    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:10.875677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:10.875647    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:12.415974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:12.415923    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:12.538344 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:12.538313    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:12.655365 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:12.655323    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:14.012939 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:14.012905    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:14.523871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:14.522794    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp" podUID="549d01c2-89c8-4c66-ac68-64857f09098b" containerName="payload-processing" containerID="cri-o://ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90" gracePeriod=25
Oct 30 03:19:14.551048 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:14.550976    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh" podUID="40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193" containerName="payload-pre-processing" containerID="cri-o://5c614d637d6525d7cbe8d675cde23ce3a4b458376ef0c04a1223fa3cebeba8c1" gracePeriod=25
Oct 30 03:19:14.892627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:14.892276    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:19:14.943819 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:14.943786    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2fc39f-d861-4c39-aa82-fd1317b358ae-kube-api-access-2xxph\" (UniqueName: \"kubernetes.io/projected/df2fc39f-d861-4c39-aa82-fd1317b358ae-kube-api-access-2xxph\") pod \"df2fc39f-d861-4c39-aa82-fd1317b358ae\" (UID: \"df2fc39f-d861-4c39-aa82-fd1317b358ae\") "
Oct 30 03:19:14.944005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:14.943953    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2fc39f-d861-4c39-aa82-fd1317b358ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2fc39f-d861-4c39-aa82-fd1317b358ae-plugins-config-volume\") pod \"df2fc39f-d861-4c39-aa82-fd1317b358ae\" (UID: \"df2fc39f-d861-4c39-aa82-fd1317b358ae\") "
Oct 30 03:19:14.944403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:14.944373    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2fc39f-d861-4c39-aa82-fd1317b358ae-plugins-config-volume" pod "df2fc39f-d861-4c39-aa82-fd1317b358ae" (UID: "df2fc39f-d861-4c39-aa82-fd1317b358ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:14.945332 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:14.944986    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2fc39f-d861-4c39-aa82-fd1317b358ae-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:14.945691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:14.945671    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:19:14.947880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:14.947841    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2fc39f-d861-4c39-aa82-fd1317b358ae-kube-api-access-2xxph" pod "df2fc39f-d861-4c39-aa82-fd1317b358ae" (UID: "df2fc39f-d861-4c39-aa82-fd1317b358ae"). InnerVolumeSpecName "kube-api-access-2xxph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:15.058871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.051791    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549d01c2-89c8-4c66-ac68-64857f09098b-kube-api-access-4r6sk\" (UniqueName: \"kubernetes.io/projected/549d01c2-89c8-4c66-ac68-64857f09098b-kube-api-access-4r6sk\") pod \"549d01c2-89c8-4c66-ac68-64857f09098b\" (UID: \"549d01c2-89c8-4c66-ac68-64857f09098b\") "
Oct 30 03:19:15.058871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.051930    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549d01c2-89c8-4c66-ac68-64857f09098b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549d01c2-89c8-4c66-ac68-64857f09098b-plugins-config-volume\") pod \"549d01c2-89c8-4c66-ac68-64857f09098b\" (UID: \"549d01c2-89c8-4c66-ac68-64857f09098b\") "
Oct 30 03:19:15.058871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.052569    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xxph\" (UniqueName: \"kubernetes.io/projected/df2fc39f-d861-4c39-aa82-fd1317b358ae-kube-api-access-2xxph\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:15.058871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.052967    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549d01c2-89c8-4c66-ac68-64857f09098b-plugins-config-volume" pod "549d01c2-89c8-4c66-ac68-64857f09098b" (UID: "549d01c2-89c8-4c66-ac68-64857f09098b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:15.124234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.124130    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549d01c2-89c8-4c66-ac68-64857f09098b-kube-api-access-4r6sk" pod "549d01c2-89c8-4c66-ac68-64857f09098b" (UID: "549d01c2-89c8-4c66-ac68-64857f09098b"). InnerVolumeSpecName "kube-api-access-4r6sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:15.162151 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.161282    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r6sk\" (UniqueName: \"kubernetes.io/projected/549d01c2-89c8-4c66-ac68-64857f09098b-kube-api-access-4r6sk\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:15.162151 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.161325    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549d01c2-89c8-4c66-ac68-64857f09098b-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:15.639618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.639582    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:15.672554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.672517    5061 generic.go:356] "Generic (PLEG): container finished" podID="549d01c2-89c8-4c66-ac68-64857f09098b" containerID="ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90" exitCode=0
Oct 30 03:19:15.672708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.672612    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp" event={"ID":"549d01c2-89c8-4c66-ac68-64857f09098b","Type":"ContainerDied","Data":"ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90"}
Oct 30 03:19:15.672708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.672679    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp" event={"ID":"549d01c2-89c8-4c66-ac68-64857f09098b","Type":"ContainerDied","Data":"53e6c19392dc5758fc62cad9d7183a25dfc2afb85fa4bd1501ca6d076e983d5b"}
Oct 30 03:19:15.672842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.672715    5061 scope.go:122] "RemoveContainer" containerID="ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90"
Oct 30 03:19:15.674503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.674343    5061 generic.go:356] "Generic (PLEG): container finished" podID="df2fc39f-d861-4c39-aa82-fd1317b358ae" containerID="aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb" exitCode=0
Oct 30 03:19:15.674503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.674406    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"
Oct 30 03:19:15.674503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.674466    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf" event={"ID":"df2fc39f-d861-4c39-aa82-fd1317b358ae","Type":"ContainerDied","Data":"aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb"}
Oct 30 03:19:15.674676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.674549    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf" event={"ID":"df2fc39f-d861-4c39-aa82-fd1317b358ae","Type":"ContainerDied","Data":"38c7e0ada430926a992ea0fb6d6a09caf3e53991fa436880d13bda95ff445314"}
Oct 30 03:19:15.677981 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.677953    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"
Oct 30 03:19:15.695359 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.695006    5061 scope.go:122] "RemoveContainer" containerID="ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90"
Oct 30 03:19:15.695359 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:19:15.695324    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90\": container with ID starting with ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90 not found: ID does not exist" containerID="ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90"
Oct 30 03:19:15.695359 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.695349    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90"} err="failed to get container status \"ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90\": rpc error: code = NotFound desc = could not find container \"ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90\": container with ID starting with ef13f390bd93e4072f47374b843f6450c5d56b1cd8f78192da1aaf4aa653ed90 not found: ID does not exist"
Oct 30 03:19:15.695555 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.695369    5061 scope.go:122] "RemoveContainer" containerID="aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb"
Oct 30 03:19:15.708480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.708459    5061 scope.go:122] "RemoveContainer" containerID="aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb"
Oct 30 03:19:15.708747 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:19:15.708718    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb\": container with ID starting with aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb not found: ID does not exist" containerID="aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb"
Oct 30 03:19:15.708839 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.708752    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb"} err="failed to get container status \"aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb\": rpc error: code = NotFound desc = could not find container \"aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb\": container with ID starting with aed450d851dcfe1d73ba39770375fa407052f9134929e1f697d2b4bcf3a36efb not found: ID does not exist"
Oct 30 03:19:15.743068 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.743045    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"]
Oct 30 03:19:15.753554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.753533    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537001-66fd974b46-cbngf"]
Oct 30 03:19:15.770381 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.770356    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:15.783971 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.783947    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"]
Oct 30 03:19:15.789099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.789077    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f21058-86d9b6f68-97sdp"]
Oct 30 03:19:15.874580 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:15.874539    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh" podUID="40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:19:16.743056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:16.743013    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:16.854106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:16.854064    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549d01c2-89c8-4c66-ac68-64857f09098b" path="/var/lib/kubelet/pods/549d01c2-89c8-4c66-ac68-64857f09098b/volumes"
Oct 30 03:19:16.854690 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:16.854666    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2fc39f-d861-4c39-aa82-fd1317b358ae" path="/var/lib/kubelet/pods/df2fc39f-d861-4c39-aa82-fd1317b358ae/volumes"
Oct 30 03:19:16.938265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:16.938234    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:17.324630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:17.324592    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:17.562749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:17.562715    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:18.230526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:18.230494    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:19.079004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:19.078962    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:20.624578 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:20.624540    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:21.817919 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:21.817887    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"]
Oct 30 03:19:21.818423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:21.818121    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd" podUID="8741b784-88ae-4230-b4a1-4184f41ab5e1" containerName="maas-api" containerID="cri-o://c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f" gracePeriod=30
Oct 30 03:19:21.904943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:21.904859    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:22.265520 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.265152    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:19:22.377380 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.377341    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-maas-api-tls\") pod \"8741b784-88ae-4230-b4a1-4184f41ab5e1\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") "
Oct 30 03:19:22.377581 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.377431    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8741b784-88ae-4230-b4a1-4184f41ab5e1-kube-api-access-whkts\" (UniqueName: \"kubernetes.io/projected/8741b784-88ae-4230-b4a1-4184f41ab5e1-kube-api-access-whkts\") pod \"8741b784-88ae-4230-b4a1-4184f41ab5e1\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") "
Oct 30 03:19:22.377581 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.377461    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8741b784-88ae-4230-b4a1-4184f41ab5e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8741b784-88ae-4230-b4a1-4184f41ab5e1-openshift-service-ca\") pod \"8741b784-88ae-4230-b4a1-4184f41ab5e1\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") "
Oct 30 03:19:22.377753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.377736    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-metrics-certs\") pod \"8741b784-88ae-4230-b4a1-4184f41ab5e1\" (UID: \"8741b784-88ae-4230-b4a1-4184f41ab5e1\") "
Oct 30 03:19:22.381284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.381255    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8741b784-88ae-4230-b4a1-4184f41ab5e1-kube-api-access-whkts" pod "8741b784-88ae-4230-b4a1-4184f41ab5e1" (UID: "8741b784-88ae-4230-b4a1-4184f41ab5e1"). InnerVolumeSpecName "kube-api-access-whkts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:22.383907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.383843    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-metrics-certs" pod "8741b784-88ae-4230-b4a1-4184f41ab5e1" (UID: "8741b784-88ae-4230-b4a1-4184f41ab5e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:22.384174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.384154    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8741b784-88ae-4230-b4a1-4184f41ab5e1-openshift-service-ca" pod "8741b784-88ae-4230-b4a1-4184f41ab5e1" (UID: "8741b784-88ae-4230-b4a1-4184f41ab5e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:22.385164 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.385145    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-maas-api-tls" pod "8741b784-88ae-4230-b4a1-4184f41ab5e1" (UID: "8741b784-88ae-4230-b4a1-4184f41ab5e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:19:22.480213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.480120    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whkts\" (UniqueName: \"kubernetes.io/projected/8741b784-88ae-4230-b4a1-4184f41ab5e1-kube-api-access-whkts\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:22.480213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.480154    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8741b784-88ae-4230-b4a1-4184f41ab5e1-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:22.480213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.480168    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:22.480213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.480181    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8741b784-88ae-4230-b4a1-4184f41ab5e1-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:22.597365 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.597325    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:22.725177 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.725131    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:22.734489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.734408    5061 generic.go:356] "Generic (PLEG): container finished" podID="8741b784-88ae-4230-b4a1-4184f41ab5e1" containerID="c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f" exitCode=0
Oct 30 03:19:22.734489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.734447    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"
Oct 30 03:19:22.734674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.734491    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd" event={"ID":"8741b784-88ae-4230-b4a1-4184f41ab5e1","Type":"ContainerDied","Data":"c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f"}
Oct 30 03:19:22.734674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.734548    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd" event={"ID":"8741b784-88ae-4230-b4a1-4184f41ab5e1","Type":"ContainerDied","Data":"7118097d3c389334e04b3118c672d74ea94aecda2d0c63e32f9d57cfea6446cd"}
Oct 30 03:19:22.734674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.734572    5061 scope.go:122] "RemoveContainer" containerID="c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f"
Oct 30 03:19:22.746400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.746381    5061 scope.go:122] "RemoveContainer" containerID="c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f"
Oct 30 03:19:22.746658 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:19:22.746642    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f\": container with ID starting with c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f not found: ID does not exist" containerID="c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f"
Oct 30 03:19:22.746704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.746665    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f"} err="failed to get container status \"c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f\": rpc error: code = NotFound desc = could not find container \"c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f\": container with ID starting with c385a1ac313e78d9c7be71402f4b9730f7851da2de963ea3543e967a30dce21f not found: ID does not exist"
Oct 30 03:19:22.783199 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.783173    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"]
Oct 30 03:19:22.807262 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.807224    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7f767-7ccbdf76d5-62mnd"]
Oct 30 03:19:22.841653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.840762    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:22.854814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:22.854781    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8741b784-88ae-4230-b4a1-4184f41ab5e1" path="/var/lib/kubelet/pods/8741b784-88ae-4230-b4a1-4184f41ab5e1/volumes"
Oct 30 03:19:23.208104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:23.208061    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:24.030545 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:24.030509    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:24.154861 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:24.154818    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:24.291204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:24.291120    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"]
Oct 30 03:19:24.291405 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:24.291382    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerName="istio-proxy" containerID="cri-o://734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8" gracePeriod=30
Oct 30 03:19:24.350748 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:24.350716    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:24.487871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:24.486560    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:24.699404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:24.699371    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:25.238962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:25.238922    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:25.449627 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:19:25.449592    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:19:25.679296 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:25.679247    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:25.840513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:25.840477    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:25.897341 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:25.897295    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh" podUID="40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:19:26.397543 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:26.397473    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:27.740108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:27.740071    5061 patch_prober.go:29] interesting pod/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:19:27.740499 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:27.740141    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:19:28.145755 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:28.145715    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:29.024277 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.024241    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:29.239293 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.239258    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:29.594094 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.594070    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:19:29.622916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.622842    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-data\") pod \"3692aa6a-073e-4ad9-9f06-7600188291d2\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") "
Oct 30 03:19:29.623065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623031    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-envoy\") pod \"3692aa6a-073e-4ad9-9f06-7600188291d2\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") "
Oct 30 03:19:29.623136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623095    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-kube-api-access-qqbsk\" (UniqueName: \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-kube-api-access-qqbsk\") pod \"3692aa6a-073e-4ad9-9f06-7600188291d2\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") "
Oct 30 03:19:29.623136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623104    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-data" pod "3692aa6a-073e-4ad9-9f06-7600188291d2" (UID: "3692aa6a-073e-4ad9-9f06-7600188291d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:29.623136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623120    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-token\" (UniqueName: \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-token\") pod \"3692aa6a-073e-4ad9-9f06-7600188291d2\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") "
Oct 30 03:19:29.623304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623138    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-certs\") pod \"3692aa6a-073e-4ad9-9f06-7600188291d2\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") "
Oct 30 03:19:29.623304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623183    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-credential-socket\") pod \"3692aa6a-073e-4ad9-9f06-7600188291d2\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") "
Oct 30 03:19:29.623304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623230    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3692aa6a-073e-4ad9-9f06-7600188291d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3692aa6a-073e-4ad9-9f06-7600188291d2-istiod-ca-cert\") pod \"3692aa6a-073e-4ad9-9f06-7600188291d2\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") "
Oct 30 03:19:29.623304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623269    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-podinfo\") pod \"3692aa6a-073e-4ad9-9f06-7600188291d2\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") "
Oct 30 03:19:29.623532 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623462    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-socket\") pod \"3692aa6a-073e-4ad9-9f06-7600188291d2\" (UID: \"3692aa6a-073e-4ad9-9f06-7600188291d2\") "
Oct 30 03:19:29.623532 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623477    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-certs" pod "3692aa6a-073e-4ad9-9f06-7600188291d2" (UID: "3692aa6a-073e-4ad9-9f06-7600188291d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:29.623532 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623508    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-credential-socket" pod "3692aa6a-073e-4ad9-9f06-7600188291d2" (UID: "3692aa6a-073e-4ad9-9f06-7600188291d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:29.623799 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623773    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3692aa6a-073e-4ad9-9f06-7600188291d2-istiod-ca-cert" pod "3692aa6a-073e-4ad9-9f06-7600188291d2" (UID: "3692aa6a-073e-4ad9-9f06-7600188291d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:29.623891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.623782    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-socket" pod "3692aa6a-073e-4ad9-9f06-7600188291d2" (UID: "3692aa6a-073e-4ad9-9f06-7600188291d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:29.624106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.624085    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.624197 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.624108    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.624197 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.624123    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3692aa6a-073e-4ad9-9f06-7600188291d2-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.624197 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.624137    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.624197 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.624146    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.625451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.625424    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-kube-api-access-qqbsk" pod "3692aa6a-073e-4ad9-9f06-7600188291d2" (UID: "3692aa6a-073e-4ad9-9f06-7600188291d2"). InnerVolumeSpecName "kube-api-access-qqbsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:29.625556 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.625520    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-podinfo" pod "3692aa6a-073e-4ad9-9f06-7600188291d2" (UID: "3692aa6a-073e-4ad9-9f06-7600188291d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:19:29.625714 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.625695    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-envoy" pod "3692aa6a-073e-4ad9-9f06-7600188291d2" (UID: "3692aa6a-073e-4ad9-9f06-7600188291d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:19:29.625763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.625734    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-token" pod "3692aa6a-073e-4ad9-9f06-7600188291d2" (UID: "3692aa6a-073e-4ad9-9f06-7600188291d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:29.725578 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.725545    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.725578 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.725576    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqbsk\" (UniqueName: \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-kube-api-access-qqbsk\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.725777 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.725589    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.725777 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.725602    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3692aa6a-073e-4ad9-9f06-7600188291d2-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:29.797186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.797153    5061 generic.go:356] "Generic (PLEG): container finished" podID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerID="734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8" exitCode=0
Oct 30 03:19:29.797384 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.797198    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"
Oct 30 03:19:29.797384 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.797237    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" event={"ID":"3692aa6a-073e-4ad9-9f06-7600188291d2","Type":"ContainerDied","Data":"734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8"}
Oct 30 03:19:29.797384 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.797304    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l" event={"ID":"3692aa6a-073e-4ad9-9f06-7600188291d2","Type":"ContainerDied","Data":"e428414c68e4889a1486770fbed274c21946f883b12aeb869fe3ab3958c96a4e"}
Oct 30 03:19:29.797384 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.797336    5061 scope.go:122] "RemoveContainer" containerID="734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8"
Oct 30 03:19:29.809275 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.809259    5061 scope.go:122] "RemoveContainer" containerID="734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8"
Oct 30 03:19:29.809538 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:19:29.809516    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8\": container with ID starting with 734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8 not found: ID does not exist" containerID="734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8"
Oct 30 03:19:29.809595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.809550    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8"} err="failed to get container status \"734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8\": rpc error: code = NotFound desc = could not find container \"734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8\": container with ID starting with 734babdea4a7d2cebb5bc799a1b0ce72492aab5886e878013ba88925e89bffc8 not found: ID does not exist"
Oct 30 03:19:29.848717 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.848640    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"]
Oct 30 03:19:29.854914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.854889    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-537001-openshift-default-74f94b549-8ws6l"]
Oct 30 03:19:29.916565 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:29.916535    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:30.038503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:30.038466    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:30.187744 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:30.187663    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:30.347869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:30.325457    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:30.838558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:30.838526    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:30.851608 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:30.851577    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" path="/var/lib/kubelet/pods/3692aa6a-073e-4ad9-9f06-7600188291d2/volumes"
Oct 30 03:19:30.954218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:30.954152    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:32.236124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:32.236083    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:33.678365 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:33.678329    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:34.303912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:34.303879    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:34.563814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:34.563721    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:34.769250 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:34.769214    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:34.898466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:34.898430    5061 generic.go:356] "Generic (PLEG): container finished" podID="40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193" containerID="5c614d637d6525d7cbe8d675cde23ce3a4b458376ef0c04a1223fa3cebeba8c1" exitCode=0
Oct 30 03:19:34.898633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:34.898498    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh" event={"ID":"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193","Type":"ContainerDied","Data":"5c614d637d6525d7cbe8d675cde23ce3a4b458376ef0c04a1223fa3cebeba8c1"}
Oct 30 03:19:34.898633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:34.898557    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh" event={"ID":"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193","Type":"ContainerDied","Data":"ac9c120e5dc265849a6a3732bfe4ef0a6a493c4348c81c6e1ca4e020c7a1aae1"}
Oct 30 03:19:34.898633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:34.898581    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac9c120e5dc265849a6a3732bfe4ef0a6a493c4348c81c6e1ca4e020c7a1aae1"
Oct 30 03:19:34.901956 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:34.901936    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:19:35.087195 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:35.087165    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-plugins-config-volume\") pod \"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193\" (UID: \"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193\") "
Oct 30 03:19:35.087401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:35.087237    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-kube-api-access-8gb5x\" (UniqueName: \"kubernetes.io/projected/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-kube-api-access-8gb5x\") pod \"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193\" (UID: \"40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193\") "
Oct 30 03:19:35.087596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:35.087573    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-plugins-config-volume" pod "40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193" (UID: "40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:35.087719 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:35.087703    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:35.089446 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:35.089419    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-kube-api-access-8gb5x" pod "40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193" (UID: "40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193"). InnerVolumeSpecName "kube-api-access-8gb5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:35.115278 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:35.115211    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:35.189769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:35.189729    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gb5x\" (UniqueName: \"kubernetes.io/projected/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193-kube-api-access-8gb5x\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:35.905427 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:35.905397    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"
Oct 30 03:19:35.957718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:35.957685    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"]
Oct 30 03:19:35.967913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:35.967886    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f21058-68466cc45f-d8kgh"]
Oct 30 03:19:36.681746 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:36.681693    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:36.849321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:36.849286    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193" path="/var/lib/kubelet/pods/40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193/volumes"
Oct 30 03:19:37.562859 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:37.562823    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:37.672423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:37.672388    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:38.481839 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:38.481807    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:38.792713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:38.792633    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:40.147938 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:40.146543    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:41.013890 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:41.013824    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:41.137256 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:41.137215    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:41.231136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:41.231104    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"]
Oct 30 03:19:41.366998 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:41.366966    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:41.494288 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:41.494231    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:41.745074 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:41.744989    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:43.783307 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:43.783261    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:45.512262 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:45.512230    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:46.231506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:46.231456    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx" podUID="0d33ad4c-62c1-4403-9fe3-832bf46c5210" containerName="payload-pre-processing" containerID="cri-o://dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7" gracePeriod=25
Oct 30 03:19:47.650907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:47.650871    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:49.385020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:49.384988    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:49.708255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:49.708164    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:50.377300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:50.377258    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx" podUID="0d33ad4c-62c1-4403-9fe3-832bf46c5210" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:19:51.639012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:51.638972    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:52.033030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:52.032930    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"]
Oct 30 03:19:52.089753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:52.089715    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"]
Oct 30 03:19:52.227817 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:52.227755    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:52.364055 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:52.364015    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:53.644725 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:53.644692    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:55.306905 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:55.306874    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:55.447325 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:19:55.447284    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:19:56.260005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.259973    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"]
Oct 30 03:19:56.260841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260816    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8741b784-88ae-4230-b4a1-4184f41ab5e1" containerName="maas-api"
Oct 30 03:19:56.260841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260837    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8741b784-88ae-4230-b4a1-4184f41ab5e1" containerName="maas-api"
Oct 30 03:19:56.260841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260858    5061 cpu_manager.go:414] "containerMap: removing container" podUID="df2fc39f-d861-4c39-aa82-fd1317b358ae" containerName="payload-processing"
Oct 30 03:19:56.260841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260863    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2fc39f-d861-4c39-aa82-fd1317b358ae" containerName="payload-processing"
Oct 30 03:19:56.260841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260870    5061 cpu_manager.go:414] "containerMap: removing container" podUID="549d01c2-89c8-4c66-ac68-64857f09098b" containerName="payload-processing"
Oct 30 03:19:56.260841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260875    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549d01c2-89c8-4c66-ac68-64857f09098b" containerName="payload-processing"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260892    5061 cpu_manager.go:414] "containerMap: removing container" podUID="40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193" containerName="payload-pre-processing"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260898    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193" containerName="payload-pre-processing"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260915    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerName="istio-proxy"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260920    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerName="istio-proxy"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260935    5061 cpu_manager.go:414] "containerMap: removing container" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerName="istio-proxy"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260940    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerName="istio-proxy"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260947    5061 cpu_manager.go:414] "containerMap: removing container" podUID="cc507dc4-2b61-4056-8ae6-0a617726004c" containerName="main"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.260952    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc507dc4-2b61-4056-8ae6-0a617726004c" containerName="main"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.261068    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="3692aa6a-073e-4ad9-9f06-7600188291d2" containerName="istio-proxy"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.261076    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc507dc4-2b61-4056-8ae6-0a617726004c" containerName="main"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.261089    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8741b784-88ae-4230-b4a1-4184f41ab5e1" containerName="maas-api"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.261096    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2fc39f-d861-4c39-aa82-fd1317b358ae" containerName="payload-processing"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.261101    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="549d01c2-89c8-4c66-ac68-64857f09098b" containerName="payload-processing"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.261109    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bb52bb-dc5f-46aa-8ebc-24e9ac5fa193" containerName="payload-pre-processing"
Oct 30 03:19:56.261263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.261117    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d65367-9c08-4b32-b9d4-002d9bc4c55c" containerName="istio-proxy"
Oct 30 03:19:56.264272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.264257    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.280927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.280904    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-credential-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.281056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.280940    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.281056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.281042    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-certs\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.281146 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.281118    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-envoy\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.281205 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.281189    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-data\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.281255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.281240    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-podinfo\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.281362 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.281347    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acdfcf7-9879-4fd9-bd26-967bcc179697-istiod-ca-cert\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.281438 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.281425    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mjf\" (UniqueName: \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-kube-api-access-z5mjf\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.281497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.281484    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-token\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.290482 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.290457    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-100ffd-openshift-default-dockercfg-db8dw\""
Oct 30 03:19:56.294831 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.294807    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"]
Oct 30 03:19:56.382280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382253    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-credential-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.382708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382286    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.382708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382427    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-certs\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.382708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382469    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-envoy\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.382708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382507    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-data\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.382708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382556    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-podinfo\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.382708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382628    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acdfcf7-9879-4fd9-bd26-967bcc179697-istiod-ca-cert\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.382708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382648    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.382708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382696    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-credential-socket\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.383166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382750    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mjf\" (UniqueName: \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-kube-api-access-z5mjf\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.383166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382797    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-token\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.383166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382805    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-certs\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.383166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.382959    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-data\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.383377 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.383359    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acdfcf7-9879-4fd9-bd26-967bcc179697-istiod-ca-cert\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.384727 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.384700    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-envoy\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.385113 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.385090    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-podinfo\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.400809 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.400778    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-token\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.401110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.401091    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mjf\" (UniqueName: \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-kube-api-access-z5mjf\") pod \"e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") " pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.585027 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.584997    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:56.615456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.615419    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:56.759438 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.759300    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:56.824532 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.824502    5061 scope.go:122] "RemoveContainer" containerID="5c614d637d6525d7cbe8d675cde23ce3a4b458376ef0c04a1223fa3cebeba8c1"
Oct 30 03:19:56.881625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.881591    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:56.937010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.936977    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"]
Oct 30 03:19:56.942505 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:19:56.942474    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acdfcf7_9879_4fd9_bd26_967bcc179697.slice/crio-39e1a26b250d46737ddedb17fb1ccf8f02c696af60a0ca7e2050232b061d80a2 WatchSource:0}: Error finding container 39e1a26b250d46737ddedb17fb1ccf8f02c696af60a0ca7e2050232b061d80a2: Status 404 returned error can't find the container with id 39e1a26b250d46737ddedb17fb1ccf8f02c696af60a0ca7e2050232b061d80a2
Oct 30 03:19:56.945781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.945710    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:56.945901 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.945835    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:56.945962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:56.945897    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:57.033981 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.033947    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n" podUID="a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b" containerName="payload-processing" containerID="cri-o://ac7c2352a7bfd79abfb791d4647d6cc04c2001bf2002c95ecac8772b361a1137" gracePeriod=25
Oct 30 03:19:57.090624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.090532    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8" podUID="60f9435d-51ae-4004-a485-3025d4333bd3" containerName="payload-pre-processing" containerID="cri-o://ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95" gracePeriod=25
Oct 30 03:19:57.203195 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.203156    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:57.381918 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.376670    5061 generic.go:356] "Generic (PLEG): container finished" podID="a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b" containerID="ac7c2352a7bfd79abfb791d4647d6cc04c2001bf2002c95ecac8772b361a1137" exitCode=0
Oct 30 03:19:57.381918 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.377071    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n" event={"ID":"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b","Type":"ContainerDied","Data":"ac7c2352a7bfd79abfb791d4647d6cc04c2001bf2002c95ecac8772b361a1137"}
Oct 30 03:19:57.402899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.400046    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" event={"ID":"3acdfcf7-9879-4fd9-bd26-967bcc179697","Type":"ContainerStarted","Data":"8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79"}
Oct 30 03:19:57.402899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.400122    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" event={"ID":"3acdfcf7-9879-4fd9-bd26-967bcc179697","Type":"ContainerStarted","Data":"39e1a26b250d46737ddedb17fb1ccf8f02c696af60a0ca7e2050232b061d80a2"}
Oct 30 03:19:57.466637 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.466578    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" podStartSLOduration=1.466559711 podStartE2EDuration="1.466559711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:57.458309987 +0000 UTC m=+2342.760488366" watchObservedRunningTime="2026-10-30 03:19:57.466559711 +0000 UTC m=+2342.768738077"
Oct 30 03:19:57.489448 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.489417    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:19:57.564355 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.564318    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-plugins-config-volume\") pod \"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b\" (UID: \"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b\") "
Oct 30 03:19:57.564519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.564429    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-kube-api-access-qc9ww\" (UniqueName: \"kubernetes.io/projected/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-kube-api-access-qc9ww\") pod \"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b\" (UID: \"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b\") "
Oct 30 03:19:57.564695 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.564669    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-plugins-config-volume" pod "a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b" (UID: "a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:19:57.565105 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.565079    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:57.566923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.566893    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-kube-api-access-qc9ww" pod "a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b" (UID: "a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b"). InnerVolumeSpecName "kube-api-access-qc9ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:19:57.586069 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.586047    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:19:57.666419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:57.666358    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc9ww\" (UniqueName: \"kubernetes.io/projected/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b-kube-api-access-qc9ww\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:19:58.441870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:58.432086    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"
Oct 30 03:19:58.442389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:58.441932    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n" event={"ID":"a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b","Type":"ContainerDied","Data":"dcd2a636abb8ca8539a283fe6d25222fac118f7ae817ef6a9ea9544b7ce3fc3f"}
Oct 30 03:19:58.442389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:58.442008    5061 scope.go:122] "RemoveContainer" containerID="ac7c2352a7bfd79abfb791d4647d6cc04c2001bf2002c95ecac8772b361a1137"
Oct 30 03:19:58.503343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:58.503311    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"]
Oct 30 03:19:58.513175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:58.513150    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2305f4-cb5597554-rbk5n"]
Oct 30 03:19:58.587169 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:58.587138    5061 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 03:19:58.587353 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:58.587197    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 03:19:58.852344 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:58.852306    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b" path="/var/lib/kubelet/pods/a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b/volumes"
Oct 30 03:19:58.879431 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:58.879392    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:59.585606 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:59.585573    5061 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 03:19:59.586018 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:19:59.585625    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 03:20:00.365172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:00.365138    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:00.377137 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:00.377101    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx" podUID="0d33ad4c-62c1-4403-9fe3-832bf46c5210" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:20:00.585960 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:00.585921    5061 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 03:20:00.586450 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:00.585982    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 03:20:01.585938 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:01.585900    5061 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 03:20:01.586118 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:01.585959    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 03:20:02.046504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:02.046471    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:02.586061 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:02.586026    5061 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 03:20:02.586263 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:02.586091    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 03:20:03.585909 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:03.585839    5061 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 03:20:03.586063 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:03.585958    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 03:20:03.845779 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:03.845693    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:04.105396 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:04.105308    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8" podUID="60f9435d-51ae-4004-a485-3025d4333bd3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:20:04.591026 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:04.590988    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:20:04.592001 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:04.591978    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:20:04.592738 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:04.592709    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:20:05.482668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:05.482632    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:06.517583 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.517558    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:20:06.535704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.535675    5061 generic.go:356] "Generic (PLEG): container finished" podID="0d33ad4c-62c1-4403-9fe3-832bf46c5210" containerID="dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7" exitCode=0
Oct 30 03:20:06.535836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.535716    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"
Oct 30 03:20:06.535836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.535754    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx" event={"ID":"0d33ad4c-62c1-4403-9fe3-832bf46c5210","Type":"ContainerDied","Data":"dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7"}
Oct 30 03:20:06.535836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.535811    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx" event={"ID":"0d33ad4c-62c1-4403-9fe3-832bf46c5210","Type":"ContainerDied","Data":"783d81c7d38c17c81f0bf03d167bbf1940a8e4cf8542b53e9c99861081b42711"}
Oct 30 03:20:06.535836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.535834    5061 scope.go:122] "RemoveContainer" containerID="dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7"
Oct 30 03:20:06.545974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.545957    5061 scope.go:122] "RemoveContainer" containerID="dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7"
Oct 30 03:20:06.546242 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:20:06.546224    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7\": container with ID starting with dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7 not found: ID does not exist" containerID="dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7"
Oct 30 03:20:06.546308 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.546248    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7"} err="failed to get container status \"dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7\": rpc error: code = NotFound desc = could not find container \"dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7\": container with ID starting with dd06c58a0972fbcbd7b52cc86be2f634baea50b2172d5ef9b8fd9da39898e6d7 not found: ID does not exist"
Oct 30 03:20:06.622669 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.622631    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d33ad4c-62c1-4403-9fe3-832bf46c5210-kube-api-access-hlsfw\" (UniqueName: \"kubernetes.io/projected/0d33ad4c-62c1-4403-9fe3-832bf46c5210-kube-api-access-hlsfw\") pod \"0d33ad4c-62c1-4403-9fe3-832bf46c5210\" (UID: \"0d33ad4c-62c1-4403-9fe3-832bf46c5210\") "
Oct 30 03:20:06.622838 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.622745    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d33ad4c-62c1-4403-9fe3-832bf46c5210-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d33ad4c-62c1-4403-9fe3-832bf46c5210-plugins-config-volume\") pod \"0d33ad4c-62c1-4403-9fe3-832bf46c5210\" (UID: \"0d33ad4c-62c1-4403-9fe3-832bf46c5210\") "
Oct 30 03:20:06.623176 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.623148    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d33ad4c-62c1-4403-9fe3-832bf46c5210-plugins-config-volume" pod "0d33ad4c-62c1-4403-9fe3-832bf46c5210" (UID: "0d33ad4c-62c1-4403-9fe3-832bf46c5210"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:06.623531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.623509    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d33ad4c-62c1-4403-9fe3-832bf46c5210-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:06.624917 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.624897    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d33ad4c-62c1-4403-9fe3-832bf46c5210-kube-api-access-hlsfw" pod "0d33ad4c-62c1-4403-9fe3-832bf46c5210" (UID: "0d33ad4c-62c1-4403-9fe3-832bf46c5210"). InnerVolumeSpecName "kube-api-access-hlsfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:06.724576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.724544    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlsfw\" (UniqueName: \"kubernetes.io/projected/0d33ad4c-62c1-4403-9fe3-832bf46c5210-kube-api-access-hlsfw\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:06.888061 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.888029    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"]
Oct 30 03:20:06.897887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:06.897864    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-100ffd-7499877555-ngfvx"]
Oct 30 03:20:07.431418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:07.431382    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:07.827267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:07.827240    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:08.770406 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:08.770375    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"]
Oct 30 03:20:08.770627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:08.770601    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws" podUID="b777b635-e2e8-427d-a843-27851fc6cbf6" containerName="maas-api" containerID="cri-o://d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b" gracePeriod=30
Oct 30 03:20:08.847100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:08.847063    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:08.851628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:08.851601    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d33ad4c-62c1-4403-9fe3-832bf46c5210" path="/var/lib/kubelet/pods/0d33ad4c-62c1-4403-9fe3-832bf46c5210/volumes"
Oct 30 03:20:09.253872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.253289    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:20:09.388441 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.388351    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b777b635-e2e8-427d-a843-27851fc6cbf6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b777b635-e2e8-427d-a843-27851fc6cbf6-openshift-service-ca\") pod \"b777b635-e2e8-427d-a843-27851fc6cbf6\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") "
Oct 30 03:20:09.388599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.388555    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b777b635-e2e8-427d-a843-27851fc6cbf6-kube-api-access-dbpk4\" (UniqueName: \"kubernetes.io/projected/b777b635-e2e8-427d-a843-27851fc6cbf6-kube-api-access-dbpk4\") pod \"b777b635-e2e8-427d-a843-27851fc6cbf6\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") "
Oct 30 03:20:09.388641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.388614    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-metrics-certs\") pod \"b777b635-e2e8-427d-a843-27851fc6cbf6\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") "
Oct 30 03:20:09.388674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.388661    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-maas-api-tls\") pod \"b777b635-e2e8-427d-a843-27851fc6cbf6\" (UID: \"b777b635-e2e8-427d-a843-27851fc6cbf6\") "
Oct 30 03:20:09.390775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.390736    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b777b635-e2e8-427d-a843-27851fc6cbf6-openshift-service-ca" pod "b777b635-e2e8-427d-a843-27851fc6cbf6" (UID: "b777b635-e2e8-427d-a843-27851fc6cbf6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:09.393050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.393024    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-maas-api-tls" pod "b777b635-e2e8-427d-a843-27851fc6cbf6" (UID: "b777b635-e2e8-427d-a843-27851fc6cbf6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:09.393816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.393788    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b777b635-e2e8-427d-a843-27851fc6cbf6-kube-api-access-dbpk4" pod "b777b635-e2e8-427d-a843-27851fc6cbf6" (UID: "b777b635-e2e8-427d-a843-27851fc6cbf6"). InnerVolumeSpecName "kube-api-access-dbpk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:09.394349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.394324    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-metrics-certs" pod "b777b635-e2e8-427d-a843-27851fc6cbf6" (UID: "b777b635-e2e8-427d-a843-27851fc6cbf6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:09.490799 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.490764    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbpk4\" (UniqueName: \"kubernetes.io/projected/b777b635-e2e8-427d-a843-27851fc6cbf6-kube-api-access-dbpk4\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:09.490799 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.490801    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:09.491007 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.490815    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777b635-e2e8-427d-a843-27851fc6cbf6-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:09.491007 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.490829    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b777b635-e2e8-427d-a843-27851fc6cbf6-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:09.569550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.569520    5061 generic.go:356] "Generic (PLEG): container finished" podID="b777b635-e2e8-427d-a843-27851fc6cbf6" containerID="d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b" exitCode=0
Oct 30 03:20:09.569732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.569568    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"
Oct 30 03:20:09.569732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.569613    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws" event={"ID":"b777b635-e2e8-427d-a843-27851fc6cbf6","Type":"ContainerDied","Data":"d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b"}
Oct 30 03:20:09.569732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.569666    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws" event={"ID":"b777b635-e2e8-427d-a843-27851fc6cbf6","Type":"ContainerDied","Data":"9536e76c6ad205a0020e41571f974314cc8de4cbf41001abcd04ad0b613ffb42"}
Oct 30 03:20:09.569732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.569694    5061 scope.go:122] "RemoveContainer" containerID="d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b"
Oct 30 03:20:09.581734 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.581709    5061 scope.go:122] "RemoveContainer" containerID="d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b"
Oct 30 03:20:09.582008 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:20:09.581987    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b\": container with ID starting with d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b not found: ID does not exist" containerID="d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b"
Oct 30 03:20:09.582086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.582017    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b"} err="failed to get container status \"d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b\": rpc error: code = NotFound desc = could not find container \"d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b\": container with ID starting with d821a1d54a15dff394420cd6f5383639c17a2f9827b08640840d2b2b7ea4e60b not found: ID does not exist"
Oct 30 03:20:09.603315 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.603294    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:09.624399 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.624378    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"]
Oct 30 03:20:09.634945 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.634920    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-100ffd-5948777cbc-6f9ws"]
Oct 30 03:20:09.733172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.733100    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:09.855820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:09.855790    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:10.231609 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:10.231568    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:10.849033 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:10.849005    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b777b635-e2e8-427d-a843-27851fc6cbf6" path="/var/lib/kubelet/pods/b777b635-e2e8-427d-a843-27851fc6cbf6/volumes"
Oct 30 03:20:11.438477 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:11.438440    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:11.859010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:11.858972    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:13.586273 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:13.586231    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:14.105305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:14.105271    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8" podUID="60f9435d-51ae-4004-a485-3025d4333bd3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:20:15.256153 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:15.256105    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:16.429964 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:16.429933    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:16.563225 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:16.563191    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:16.728046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:16.727949    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"]
Oct 30 03:20:16.728270 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:16.728242    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerName="istio-proxy" containerID="cri-o://8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79" gracePeriod=30
Oct 30 03:20:16.795591 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:16.795552    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:16.940933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:16.940895    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:17.362994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.362941    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:20:17.408526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.408496    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f9435d-51ae-4004-a485-3025d4333bd3-kube-api-access-7ctmq\" (UniqueName: \"kubernetes.io/projected/60f9435d-51ae-4004-a485-3025d4333bd3-kube-api-access-7ctmq\") pod \"60f9435d-51ae-4004-a485-3025d4333bd3\" (UID: \"60f9435d-51ae-4004-a485-3025d4333bd3\") "
Oct 30 03:20:17.408715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.408700    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f9435d-51ae-4004-a485-3025d4333bd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f9435d-51ae-4004-a485-3025d4333bd3-plugins-config-volume\") pod \"60f9435d-51ae-4004-a485-3025d4333bd3\" (UID: \"60f9435d-51ae-4004-a485-3025d4333bd3\") "
Oct 30 03:20:17.409068 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.409043    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f9435d-51ae-4004-a485-3025d4333bd3-plugins-config-volume" pod "60f9435d-51ae-4004-a485-3025d4333bd3" (UID: "60f9435d-51ae-4004-a485-3025d4333bd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:17.409310 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.409293    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f9435d-51ae-4004-a485-3025d4333bd3-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:17.410736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.410704    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f9435d-51ae-4004-a485-3025d4333bd3-kube-api-access-7ctmq" pod "60f9435d-51ae-4004-a485-3025d4333bd3" (UID: "60f9435d-51ae-4004-a485-3025d4333bd3"). InnerVolumeSpecName "kube-api-access-7ctmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:17.510543 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.510509    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ctmq\" (UniqueName: \"kubernetes.io/projected/60f9435d-51ae-4004-a485-3025d4333bd3-kube-api-access-7ctmq\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:17.631349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.631251    5061 generic.go:356] "Generic (PLEG): container finished" podID="60f9435d-51ae-4004-a485-3025d4333bd3" containerID="ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95" exitCode=0
Oct 30 03:20:17.631349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.631309    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"
Oct 30 03:20:17.631349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.631324    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8" event={"ID":"60f9435d-51ae-4004-a485-3025d4333bd3","Type":"ContainerDied","Data":"ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95"}
Oct 30 03:20:17.631644 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.631379    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8" event={"ID":"60f9435d-51ae-4004-a485-3025d4333bd3","Type":"ContainerDied","Data":"789097d1c704fc12ab462e7bd5eee8972f171b949756ab0f9bc5a88afa608502"}
Oct 30 03:20:17.631644 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.631403    5061 scope.go:122] "RemoveContainer" containerID="ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95"
Oct 30 03:20:17.642364 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.642346    5061 scope.go:122] "RemoveContainer" containerID="ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95"
Oct 30 03:20:17.642606 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:20:17.642589    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95\": container with ID starting with ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95 not found: ID does not exist" containerID="ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95"
Oct 30 03:20:17.642660 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.642614    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95"} err="failed to get container status \"ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95\": rpc error: code = NotFound desc = could not find container \"ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95\": container with ID starting with ff0c5c13c5aa49b4f90a3e1b7cf62b027e402ecbbfb4af64f010424cfc4f7c95 not found: ID does not exist"
Oct 30 03:20:17.677911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.677888    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"]
Oct 30 03:20:17.688419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:17.688395    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2305f4-8478986cbc-p5nx8"]
Oct 30 03:20:18.639447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:18.639412    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:18.849004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:18.848968    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f9435d-51ae-4004-a485-3025d4333bd3" path="/var/lib/kubelet/pods/60f9435d-51ae-4004-a485-3025d4333bd3/volumes"
Oct 30 03:20:19.594540 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:19.594506    5061 patch_prober.go:29] interesting pod/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:20:19.594707 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:19.594573    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:20:20.171982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:20.171947    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:21.600715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:21.600678    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:22.143913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.143882    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:20:22.245398 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.245366    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:22.286546 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.286471    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-kube-api-access-z5mjf\" (UniqueName: \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-kube-api-access-z5mjf\") pod \"3acdfcf7-9879-4fd9-bd26-967bcc179697\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") "
Oct 30 03:20:22.286546 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.286514    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-data\") pod \"3acdfcf7-9879-4fd9-bd26-967bcc179697\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") "
Oct 30 03:20:22.286768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.286632    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-credential-socket\") pod \"3acdfcf7-9879-4fd9-bd26-967bcc179697\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") "
Oct 30 03:20:22.286768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.286723    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-token\" (UniqueName: \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-token\") pod \"3acdfcf7-9879-4fd9-bd26-967bcc179697\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") "
Oct 30 03:20:22.286911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.286784    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-certs\") pod \"3acdfcf7-9879-4fd9-bd26-967bcc179697\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") "
Oct 30 03:20:22.286911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.286812    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-data" pod "3acdfcf7-9879-4fd9-bd26-967bcc179697" (UID: "3acdfcf7-9879-4fd9-bd26-967bcc179697"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.286911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.286828    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3acdfcf7-9879-4fd9-bd26-967bcc179697-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acdfcf7-9879-4fd9-bd26-967bcc179697-istiod-ca-cert\") pod \"3acdfcf7-9879-4fd9-bd26-967bcc179697\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") "
Oct 30 03:20:22.287095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.286915    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-credential-socket" pod "3acdfcf7-9879-4fd9-bd26-967bcc179697" (UID: "3acdfcf7-9879-4fd9-bd26-967bcc179697"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.287095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.286939    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-envoy\") pod \"3acdfcf7-9879-4fd9-bd26-967bcc179697\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") "
Oct 30 03:20:22.287095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.287068    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-certs" pod "3acdfcf7-9879-4fd9-bd26-967bcc179697" (UID: "3acdfcf7-9879-4fd9-bd26-967bcc179697"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.287095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.287082    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-socket\") pod \"3acdfcf7-9879-4fd9-bd26-967bcc179697\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") "
Oct 30 03:20:22.287324 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.287122    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-podinfo\") pod \"3acdfcf7-9879-4fd9-bd26-967bcc179697\" (UID: \"3acdfcf7-9879-4fd9-bd26-967bcc179697\") "
Oct 30 03:20:22.287324 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.287143    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acdfcf7-9879-4fd9-bd26-967bcc179697-istiod-ca-cert" pod "3acdfcf7-9879-4fd9-bd26-967bcc179697" (UID: "3acdfcf7-9879-4fd9-bd26-967bcc179697"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:22.287436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.287342    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-socket" pod "3acdfcf7-9879-4fd9-bd26-967bcc179697" (UID: "3acdfcf7-9879-4fd9-bd26-967bcc179697"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.287681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.287659    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.287681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.287680    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.287895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.287693    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.287895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.287705    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.287895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.287714    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3acdfcf7-9879-4fd9-bd26-967bcc179697-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.289095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.289061    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-kube-api-access-z5mjf" pod "3acdfcf7-9879-4fd9-bd26-967bcc179697" (UID: "3acdfcf7-9879-4fd9-bd26-967bcc179697"). InnerVolumeSpecName "kube-api-access-z5mjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:22.289208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.289187    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-envoy" pod "3acdfcf7-9879-4fd9-bd26-967bcc179697" (UID: "3acdfcf7-9879-4fd9-bd26-967bcc179697"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:22.289415 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.289398    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-token" pod "3acdfcf7-9879-4fd9-bd26-967bcc179697" (UID: "3acdfcf7-9879-4fd9-bd26-967bcc179697"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:22.289515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.289501    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-podinfo" pod "3acdfcf7-9879-4fd9-bd26-967bcc179697" (UID: "3acdfcf7-9879-4fd9-bd26-967bcc179697"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:20:22.388837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.388785    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.388837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.388842    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.389121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.388870    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3acdfcf7-9879-4fd9-bd26-967bcc179697-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.389121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.388879    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mjf\" (UniqueName: \"kubernetes.io/projected/3acdfcf7-9879-4fd9-bd26-967bcc179697-kube-api-access-z5mjf\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:22.474645 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.474609    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:22.608517 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.608487    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:22.669173 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.669141    5061 generic.go:356] "Generic (PLEG): container finished" podID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerID="8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79" exitCode=0
Oct 30 03:20:22.669349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.669194    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"
Oct 30 03:20:22.669349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.669221    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" event={"ID":"3acdfcf7-9879-4fd9-bd26-967bcc179697","Type":"ContainerDied","Data":"8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79"}
Oct 30 03:20:22.669349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.669270    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b" event={"ID":"3acdfcf7-9879-4fd9-bd26-967bcc179697","Type":"ContainerDied","Data":"39e1a26b250d46737ddedb17fb1ccf8f02c696af60a0ca7e2050232b061d80a2"}
Oct 30 03:20:22.669349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.669295    5061 scope.go:122] "RemoveContainer" containerID="8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79"
Oct 30 03:20:22.702969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.702931    5061 scope.go:122] "RemoveContainer" containerID="8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79"
Oct 30 03:20:22.704570 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:20:22.704340    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79\": container with ID starting with 8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79 not found: ID does not exist" containerID="8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79"
Oct 30 03:20:22.704570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.704407    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79"} err="failed to get container status \"8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79\": rpc error: code = NotFound desc = could not find container \"8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79\": container with ID starting with 8585f2c232c10b63bd43cc92a672a2515e7908433b8e20973a38cdd5d9844c79 not found: ID does not exist"
Oct 30 03:20:22.759870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.759430    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:22.759870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.759524    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"]
Oct 30 03:20:22.781873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.777158    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-100ffd-openshift-default-5746c55455-t4v7b"]
Oct 30 03:20:22.848609 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:22.848575    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" path="/var/lib/kubelet/pods/3acdfcf7-9879-4fd9-bd26-967bcc179697/volumes"
Oct 30 03:20:23.057251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:23.057221    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:23.197412 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:23.197382    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:23.702747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:23.702712    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:23.884911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:23.883478    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:24.676625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:24.676591    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:25.448352 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:20:25.448318    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:20:25.813790 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:25.813753    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:26.741977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:26.741929    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:27.572448 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:27.572412    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:27.775545 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:27.775512    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:28.285411 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:28.285360    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:28.471519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:28.471464    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:29.402912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:29.402838    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:30.884627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:30.884594    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:31.495974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:31.495931    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:31.655699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:31.655652    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:31.831956 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:31.831923    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"]
Oct 30 03:20:31.832195 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:31.832169    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" podUID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" containerName="istio-proxy" containerID="cri-o://cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7" gracePeriod=30
Oct 30 03:20:31.900716 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:31.900688    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:32.676997 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:32.676962    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:33.384474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:33.384441    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:33.524465 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:33.524425    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:35.012840 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:35.012807    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:36.733157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:36.733125    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:36.887592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:36.887558    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:37.031814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.031626    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:37.268959 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.268935    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:20:37.305713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.305688    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-token\" (UniqueName: \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-token\") pod \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") "
Oct 30 03:20:37.305878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.305761    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-data\") pod \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") "
Oct 30 03:20:37.305878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.305785    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-certs\") pod \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") "
Oct 30 03:20:37.305878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.305820    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-podinfo\") pod \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") "
Oct 30 03:20:37.306107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.305888    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-socket\") pod \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") "
Oct 30 03:20:37.306107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.305937    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-envoy\") pod \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") "
Oct 30 03:20:37.306107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.305964    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istiod-ca-cert\") pod \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") "
Oct 30 03:20:37.306279 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306139    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-credential-socket\") pod \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") "
Oct 30 03:20:37.306279 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306145    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-certs" pod "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" (UID: "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.306279 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306184    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-kube-api-access-x52gh\" (UniqueName: \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-kube-api-access-x52gh\") pod \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\" (UID: \"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2\") "
Oct 30 03:20:37.306896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306183    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-data" pod "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" (UID: "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.306896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306213    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-socket" pod "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" (UID: "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.306896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306405    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-credential-socket" pod "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" (UID: "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.306896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306514    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istiod-ca-cert" pod "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" (UID: "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:37.306896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306891    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.307213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306914    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.307213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306931    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.307213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306944    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.307213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.306958    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.308149 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.308125    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-podinfo" pod "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" (UID: "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:20:37.308349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.308329    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-envoy" pod "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" (UID: "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:37.308431 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.308396    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-token" pod "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" (UID: "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:37.308482 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.308425    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-kube-api-access-x52gh" pod "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" (UID: "8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2"). InnerVolumeSpecName "kube-api-access-x52gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:37.408101 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.408072    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x52gh\" (UniqueName: \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-kube-api-access-x52gh\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.408101 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.408099    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.408270 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.408109    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.408270 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.408117    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:20:37.876977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.876936    5061 generic.go:356] "Generic (PLEG): container finished" podID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" containerID="cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7" exitCode=0
Oct 30 03:20:37.876977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.876982    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" event={"ID":"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2","Type":"ContainerDied","Data":"cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7"}
Oct 30 03:20:37.877591 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.876991    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"
Oct 30 03:20:37.877591 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.877030    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh" event={"ID":"8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2","Type":"ContainerDied","Data":"ed1a04dfbde976670604cdb7be66007f524183f3bd8575efda5f9b70dc12c2e7"}
Oct 30 03:20:37.877591 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.877056    5061 scope.go:122] "RemoveContainer" containerID="cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7"
Oct 30 03:20:37.898546 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.898515    5061 scope.go:122] "RemoveContainer" containerID="cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7"
Oct 30 03:20:37.898899 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:20:37.898875    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7\": container with ID starting with cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7 not found: ID does not exist" containerID="cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7"
Oct 30 03:20:37.898987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.898908    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7"} err="failed to get container status \"cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7\": rpc error: code = NotFound desc = could not find container \"cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7\": container with ID starting with cf7f74be57b9f8169da09d353806419da585d1f77e95e61185549a9b405bdac7 not found: ID does not exist"
Oct 30 03:20:37.941349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.941316    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"]
Oct 30 03:20:37.951478 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:37.951446    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2305f4-openshift-default-7f9d57bfdf-xvwhh"]
Oct 30 03:20:38.665651 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:38.665613    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:38.829555 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:38.829521    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:38.849814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:38.849781    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" path="/var/lib/kubelet/pods/8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2/volumes"
Oct 30 03:20:39.161803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:39.161767    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:41.372735 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:41.372703    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:42.328625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:42.328587    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:43.017036 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:43.017002    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:43.870362 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:43.870312    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:44.840748 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:44.840708    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:46.692695 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:46.692663    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:48.249643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:48.249608    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:49.402219 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:49.402164    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:49.538491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:49.538454    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:49.882467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:49.882437    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:50.211895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:50.211772    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:50.573048 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:50.573015    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:50.968635 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:50.968565    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:51.133637 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:51.133601    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:51.287334 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:51.287237    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:52.143704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:52.143673    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:53.992965 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:53.992930    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:54.228484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:54.228443    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:54.499141 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:54.499108    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:55.340830 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:55.340801    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:20:55.343795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:55.343774    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:20:55.344452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:55.344437    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:20:55.347656 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:55.347639    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:20:55.448784 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:20:55.448753    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:20:56.102366 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.102331    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:56.820052 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.820000    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:56.945218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945194    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"]
Oct 30 03:20:56.945881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945863    5061 cpu_manager.go:414] "containerMap: removing container" podUID="60f9435d-51ae-4004-a485-3025d4333bd3" containerName="payload-pre-processing"
Oct 30 03:20:56.945940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945881    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f9435d-51ae-4004-a485-3025d4333bd3" containerName="payload-pre-processing"
Oct 30 03:20:56.945940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945905    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" containerName="istio-proxy"
Oct 30 03:20:56.945940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945912    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" containerName="istio-proxy"
Oct 30 03:20:56.945940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945934    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0d33ad4c-62c1-4403-9fe3-832bf46c5210" containerName="payload-pre-processing"
Oct 30 03:20:56.945940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945939    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d33ad4c-62c1-4403-9fe3-832bf46c5210" containerName="payload-pre-processing"
Oct 30 03:20:56.946110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945951    5061 cpu_manager.go:414] "containerMap: removing container" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerName="istio-proxy"
Oct 30 03:20:56.946110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945956    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerName="istio-proxy"
Oct 30 03:20:56.946110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945971    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b777b635-e2e8-427d-a843-27851fc6cbf6" containerName="maas-api"
Oct 30 03:20:56.946110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945976    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b777b635-e2e8-427d-a843-27851fc6cbf6" containerName="maas-api"
Oct 30 03:20:56.946110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945986    5061 cpu_manager.go:414] "containerMap: removing container" podUID="a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b" containerName="payload-processing"
Oct 30 03:20:56.946110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.945991    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b" containerName="payload-processing"
Oct 30 03:20:56.946110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.946101    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d33ad4c-62c1-4403-9fe3-832bf46c5210" containerName="payload-pre-processing"
Oct 30 03:20:56.946317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.946113    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="3acdfcf7-9879-4fd9-bd26-967bcc179697" containerName="istio-proxy"
Oct 30 03:20:56.946317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.946121    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b777b635-e2e8-427d-a843-27851fc6cbf6" containerName="maas-api"
Oct 30 03:20:56.946317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.946128    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74faa5d-6ae5-42ed-adb2-6b0cc14cfa8b" containerName="payload-processing"
Oct 30 03:20:56.946317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.946136    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f9435d-51ae-4004-a485-3025d4333bd3" containerName="payload-pre-processing"
Oct 30 03:20:56.946317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.946148    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ddfe1b3-af75-46ac-a5a1-5ece8c06ede2" containerName="istio-proxy"
Oct 30 03:20:56.951242 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.951224    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:20:56.952452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.952425    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:56.970801 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.970778    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"]
Oct 30 03:20:56.976478 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.976456    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-734274\""
Oct 30 03:20:56.976618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:56.976502    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-734274-dockercfg-bxwzz\""
Oct 30 03:20:57.078334 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.078147    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46f4\" (UniqueName: \"kubernetes.io/projected/e8af7a5c-9090-4033-b2d6-4acaa05490c6-kube-api-access-z46f4\") pod \"payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9\" (UID: \"e8af7a5c-9090-4033-b2d6-4acaa05490c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:20:57.078999 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.078958    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8af7a5c-9090-4033-b2d6-4acaa05490c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9\" (UID: \"e8af7a5c-9090-4033-b2d6-4acaa05490c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:20:57.132759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.132713    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"]
Oct 30 03:20:57.175072 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.175039    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"]
Oct 30 03:20:57.175401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.175380    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:20:57.182562 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.182144    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1805274d-4b09-4dad-ade2-1e42d4caf20f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5\" (UID: \"1805274d-4b09-4dad-ade2-1e42d4caf20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:20:57.184345 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.182896    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46f4\" (UniqueName: \"kubernetes.io/projected/e8af7a5c-9090-4033-b2d6-4acaa05490c6-kube-api-access-z46f4\") pod \"payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9\" (UID: \"e8af7a5c-9090-4033-b2d6-4acaa05490c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:20:57.184345 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.183035    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8af7a5c-9090-4033-b2d6-4acaa05490c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9\" (UID: \"e8af7a5c-9090-4033-b2d6-4acaa05490c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:20:57.184345 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.183107    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmclt\" (UniqueName: \"kubernetes.io/projected/1805274d-4b09-4dad-ade2-1e42d4caf20f-kube-api-access-gmclt\") pod \"payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5\" (UID: \"1805274d-4b09-4dad-ade2-1e42d4caf20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:20:57.184345 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.184279    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8af7a5c-9090-4033-b2d6-4acaa05490c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9\" (UID: \"e8af7a5c-9090-4033-b2d6-4acaa05490c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:20:57.234615 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.234578    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46f4\" (UniqueName: \"kubernetes.io/projected/e8af7a5c-9090-4033-b2d6-4acaa05490c6-kube-api-access-z46f4\") pod \"payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9\" (UID: \"e8af7a5c-9090-4033-b2d6-4acaa05490c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:20:57.266151 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.266123    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:20:57.284631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.284598    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmclt\" (UniqueName: \"kubernetes.io/projected/1805274d-4b09-4dad-ade2-1e42d4caf20f-kube-api-access-gmclt\") pod \"payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5\" (UID: \"1805274d-4b09-4dad-ade2-1e42d4caf20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:20:57.284764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.284697    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1805274d-4b09-4dad-ade2-1e42d4caf20f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5\" (UID: \"1805274d-4b09-4dad-ade2-1e42d4caf20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:20:57.285411 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.285389    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1805274d-4b09-4dad-ade2-1e42d4caf20f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5\" (UID: \"1805274d-4b09-4dad-ade2-1e42d4caf20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:20:57.305121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.305098    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmclt\" (UniqueName: \"kubernetes.io/projected/1805274d-4b09-4dad-ade2-1e42d4caf20f-kube-api-access-gmclt\") pod \"payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5\" (UID: \"1805274d-4b09-4dad-ade2-1e42d4caf20f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:20:57.531751 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.531721    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:20:57.581597 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.581502    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"]
Oct 30 03:20:57.586267 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:20:57.586239    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8af7a5c_9090_4033_b2d6_4acaa05490c6.slice/crio-8ea97f3362f0d5db52410cdcb030d96b19134b302349e054fb86f79d29e408f6 WatchSource:0}: Error finding container 8ea97f3362f0d5db52410cdcb030d96b19134b302349e054fb86f79d29e408f6: Status 404 returned error can't find the container with id 8ea97f3362f0d5db52410cdcb030d96b19134b302349e054fb86f79d29e408f6
Oct 30 03:20:57.843006 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:57.842981    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"]
Oct 30 03:20:57.845551 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:20:57.845523    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1805274d_4b09_4dad_ade2_1e42d4caf20f.slice/crio-427e35a14c9bfce53e8342da219fd1e9397331afcb12c16283f81a8ed368eda3 WatchSource:0}: Error finding container 427e35a14c9bfce53e8342da219fd1e9397331afcb12c16283f81a8ed368eda3: Status 404 returned error can't find the container with id 427e35a14c9bfce53e8342da219fd1e9397331afcb12c16283f81a8ed368eda3
Oct 30 03:20:58.140929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.140834    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5" event={"ID":"1805274d-4b09-4dad-ade2-1e42d4caf20f","Type":"ContainerStarted","Data":"9e7a5eb08ca9bc43855a26ea412785d55e85fbb898ee5ecbeae6deffdce825dd"}
Oct 30 03:20:58.141088 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.140937    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:20:58.141088 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.140956    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5" event={"ID":"1805274d-4b09-4dad-ade2-1e42d4caf20f","Type":"ContainerStarted","Data":"427e35a14c9bfce53e8342da219fd1e9397331afcb12c16283f81a8ed368eda3"}
Oct 30 03:20:58.142411 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.142382    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9" event={"ID":"e8af7a5c-9090-4033-b2d6-4acaa05490c6","Type":"ContainerStarted","Data":"806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471"}
Oct 30 03:20:58.142519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.142432    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:20:58.142519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.142450    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9" event={"ID":"e8af7a5c-9090-4033-b2d6-4acaa05490c6","Type":"ContainerStarted","Data":"8ea97f3362f0d5db52410cdcb030d96b19134b302349e054fb86f79d29e408f6"}
Oct 30 03:20:58.186186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.186143    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5" podStartSLOduration=1.1861300909999999 podStartE2EDuration="1.186130091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:58.182264853 +0000 UTC m=+2403.484443229" watchObservedRunningTime="2026-10-30 03:20:58.186130091 +0000 UTC m=+2403.488308488"
Oct 30 03:20:58.221302 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.221254    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9" podStartSLOduration=2.221241173 podStartE2EDuration="2.221241173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:58.218168177 +0000 UTC m=+2403.520346542" watchObservedRunningTime="2026-10-30 03:20:58.221241173 +0000 UTC m=+2403.523419537"
Oct 30 03:20:58.266711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.266682    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:58.594231 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.594192    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"]
Oct 30 03:20:58.598935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.598908    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.612475 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.612456    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-9d3552-openshift-default-dockercfg-k8rl8\""
Oct 30 03:20:58.627160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.627133    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"]
Oct 30 03:20:58.712408 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.712372    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgjft\" (UniqueName: \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-kube-api-access-xgjft\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.712576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.712421    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-envoy\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.712576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.712537    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.712663 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.712587    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-token\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.712754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.712734    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-credential-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.712811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.712787    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-podinfo\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.712907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.712868    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-data\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.712977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.712958    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-certs\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.713057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.713043    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.813834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.813802    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-credential-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.813876    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-podinfo\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.813911    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-data\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.813950    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-certs\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.814024    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.814065    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjft\" (UniqueName: \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-kube-api-access-xgjft\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.814093    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-envoy\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.814183    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.814239    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-credential-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.814284    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-data\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.814448    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-certs\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.814495    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-token\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.814717 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.814611    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-socket\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.815361 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.815337    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.816518 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.816492    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-envoy\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.816620 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.816603    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-podinfo\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.832290 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.832267    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-token\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.832394 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.832367    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjft\" (UniqueName: \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-kube-api-access-xgjft\") pod \"e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") " pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:58.914947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:58.914841    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:20:59.240476 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:59.240453    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"]
Oct 30 03:20:59.243322 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:20:59.243291    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bce83b4_dca3_4ad5_a05e_940006d42ac3.slice/crio-526e7742811201115ada288e974ab32cdc0bc110825102ff928647c244070cab WatchSource:0}: Error finding container 526e7742811201115ada288e974ab32cdc0bc110825102ff928647c244070cab: Status 404 returned error can't find the container with id 526e7742811201115ada288e974ab32cdc0bc110825102ff928647c244070cab
Oct 30 03:20:59.246325 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:59.246288    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:20:59.246449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:59.246380    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:20:59.246449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:20:59.246427    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:00.161193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:00.161148    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9" event={"ID":"1bce83b4-dca3-4ad5-a05e-940006d42ac3","Type":"ContainerStarted","Data":"875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42"}
Oct 30 03:21:00.161673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:00.161217    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9" event={"ID":"1bce83b4-dca3-4ad5-a05e-940006d42ac3","Type":"ContainerStarted","Data":"526e7742811201115ada288e974ab32cdc0bc110825102ff928647c244070cab"}
Oct 30 03:21:00.292879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:00.291465    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:00.296447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:00.296330    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9" podStartSLOduration=2.296314916 podStartE2EDuration="2.296314916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:00.290041601 +0000 UTC m=+2405.592219992" watchObservedRunningTime="2026-10-30 03:21:00.296314916 +0000 UTC m=+2405.598493281"
Oct 30 03:21:00.915921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:00.915886    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:21:00.917218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:00.917189    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 03:21:00.917347 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:00.917243    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9" podUID="1bce83b4-dca3-4ad5-a05e-940006d42ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 03:21:01.915511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:01.915471    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 03:21:01.915944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:01.915532    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9" podUID="1bce83b4-dca3-4ad5-a05e-940006d42ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 03:21:02.245213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:02.245110    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:02.915899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:02.915837    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 03:21:02.916320 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:02.915924    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9" podUID="1bce83b4-dca3-4ad5-a05e-940006d42ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 03:21:02.917894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:02.917868    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"]
Oct 30 03:21:02.918085 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:02.918065    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" containerName="istio-proxy" containerID="cri-o://5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419" gracePeriod=30
Oct 30 03:21:03.783535 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:03.783467    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:03.915723 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:03.915688    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 03:21:03.915926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:03.915742    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9" podUID="1bce83b4-dca3-4ad5-a05e-940006d42ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 03:21:04.548913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.548874    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"]
Oct 30 03:21:04.549098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.549075    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy" containerID="cri-o://88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e" gracePeriod=30
Oct 30 03:21:04.915757 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.915665    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 03:21:04.915757 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.915728    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9" podUID="1bce83b4-dca3-4ad5-a05e-940006d42ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 03:21:04.946599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.946551    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"]
Oct 30 03:21:04.951579 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.951551    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:04.968491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.968465    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"]
Oct 30 03:21:04.993875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.993836    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:04.994010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.993902    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94dba308-852e-4767-a793-5a2e82b13a3e-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:04.994064 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.994032    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:04.994064 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.994060    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94dba308-852e-4767-a793-5a2e82b13a3e-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:04.994179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.994164    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:04.994218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.994192    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:04.994261 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.994224    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqff\" (UniqueName: \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-kube-api-access-6jqff\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:04.994299 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.994276    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:04.994376 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:04.994362    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.095806 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.095767    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.095814    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.095888    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqff\" (UniqueName: \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-kube-api-access-6jqff\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.095910    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.096011    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.096054    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.096122    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94dba308-852e-4767-a793-5a2e82b13a3e-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.096194    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.096228    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94dba308-852e-4767-a793-5a2e82b13a3e-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.096271    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.096336    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.096362    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.096556 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.096531    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.097044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.097025    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94dba308-852e-4767-a793-5a2e82b13a3e-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.098380 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.098334    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.098716 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.098695    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94dba308-852e-4767-a793-5a2e82b13a3e-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.113625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.113601    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.113972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.113944    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqff\" (UniqueName: \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-kube-api-access-6jqff\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.266539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.266439    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:05.348296 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.348220    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:05.497293 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.494890    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:05.623127 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.623098    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"]
Oct 30 03:21:05.625907 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:21:05.625877    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94dba308_852e_4767_a793_5a2e82b13a3e.slice/crio-becf31d528f20f8f96a2876dfb8954f527b047b96224f864fc7eba1b4574a901 WatchSource:0}: Error finding container becf31d528f20f8f96a2876dfb8954f527b047b96224f864fc7eba1b4574a901: Status 404 returned error can't find the container with id becf31d528f20f8f96a2876dfb8954f527b047b96224f864fc7eba1b4574a901
Oct 30 03:21:05.628901 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.628843    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:05.629005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.628941    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:05.629005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.628973    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:05.921751 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.921674    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:21:05.922398 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.922372    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:21:05.923475 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:05.923448    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:21:06.224688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:06.224496    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" event={"ID":"94dba308-852e-4767-a793-5a2e82b13a3e","Type":"ContainerStarted","Data":"478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d"}
Oct 30 03:21:06.225022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:06.224996    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" event={"ID":"94dba308-852e-4767-a793-5a2e82b13a3e","Type":"ContainerStarted","Data":"becf31d528f20f8f96a2876dfb8954f527b047b96224f864fc7eba1b4574a901"}
Oct 30 03:21:06.267480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:06.267446    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:06.269774 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:06.269734    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 03:21:06.269974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:06.269806    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" podUID="94dba308-852e-4767-a793-5a2e82b13a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 03:21:06.275246 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:06.275186    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" podStartSLOduration=2.275168574 podStartE2EDuration="2.275168574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:06.270821358 +0000 UTC m=+2411.572999720" watchObservedRunningTime="2026-10-30 03:21:06.275168574 +0000 UTC m=+2411.577346941"
Oct 30 03:21:06.843873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:06.839378    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:06.967350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:06.967317    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:07.267397 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:07.267314    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 03:21:07.267397 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:07.267378    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" podUID="94dba308-852e-4767-a793-5a2e82b13a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 03:21:07.291647 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:07.291618    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:07.786359 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:07.786328    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:07.918280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:07.917802    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:08.325109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.324996    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 03:21:08.325723 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.325675    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" podUID="94dba308-852e-4767-a793-5a2e82b13a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 03:21:08.389406 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.389368    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 03:21:08.400876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.393332    5061 generic.go:356] "Generic (PLEG): container finished" podID="2bd23c84-3409-4a96-b587-5d8270a17205" containerID="5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419" exitCode=0
Oct 30 03:21:08.400876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.394637    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" event={"ID":"2bd23c84-3409-4a96-b587-5d8270a17205","Type":"ContainerDied","Data":"5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419"}
Oct 30 03:21:08.400876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.394690    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn" event={"ID":"2bd23c84-3409-4a96-b587-5d8270a17205","Type":"ContainerDied","Data":"90889e7bff0242e15dec3fede20fa93686715dee6c7e91249648e019ee28bfd6"}
Oct 30 03:21:08.400876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.394721    5061 scope.go:122] "RemoveContainer" containerID="5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419"
Oct 30 03:21:08.497320 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.497300    5061 scope.go:122] "RemoveContainer" containerID="5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419"
Oct 30 03:21:08.497834 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:21:08.497810    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419\": container with ID starting with 5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419 not found: ID does not exist" containerID="5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419"
Oct 30 03:21:08.498434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.498406    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419"} err="failed to get container status \"5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419\": rpc error: code = NotFound desc = could not find container \"5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419\": container with ID starting with 5cbf91d4c3354da6af0449da08042998b568048d48ee6759f19898160bd95419 not found: ID does not exist"
Oct 30 03:21:08.518704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.518676    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bd23c84-3409-4a96-b587-5d8270a17205-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd23c84-3409-4a96-b587-5d8270a17205-istio-podinfo\") pod \"2bd23c84-3409-4a96-b587-5d8270a17205\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") "
Oct 30 03:21:08.518891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.518722    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-socket\") pod \"2bd23c84-3409-4a96-b587-5d8270a17205\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") "
Oct 30 03:21:08.533871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.530944    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-socket" pod "2bd23c84-3409-4a96-b587-5d8270a17205" (UID: "2bd23c84-3409-4a96-b587-5d8270a17205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:08.546626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.542958    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-istio-token\" (UniqueName: \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-istio-token\") pod \"2bd23c84-3409-4a96-b587-5d8270a17205\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") "
Oct 30 03:21:08.546626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.543099    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-certs\") pod \"2bd23c84-3409-4a96-b587-5d8270a17205\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") "
Oct 30 03:21:08.546626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.543156    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-kube-api-access-m4ddg\" (UniqueName: \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-kube-api-access-m4ddg\") pod \"2bd23c84-3409-4a96-b587-5d8270a17205\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") "
Oct 30 03:21:08.546626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.543189    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd23c84-3409-4a96-b587-5d8270a17205-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd23c84-3409-4a96-b587-5d8270a17205-istiod-ca-cert\") pod \"2bd23c84-3409-4a96-b587-5d8270a17205\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") "
Oct 30 03:21:08.546626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.543347    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-envoy\") pod \"2bd23c84-3409-4a96-b587-5d8270a17205\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") "
Oct 30 03:21:08.546626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.543389    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-credential-socket\") pod \"2bd23c84-3409-4a96-b587-5d8270a17205\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") "
Oct 30 03:21:08.546626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.543427    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-data\") pod \"2bd23c84-3409-4a96-b587-5d8270a17205\" (UID: \"2bd23c84-3409-4a96-b587-5d8270a17205\") "
Oct 30 03:21:08.546626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.544312    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:08.549875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.547643    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-istio-token" pod "2bd23c84-3409-4a96-b587-5d8270a17205" (UID: "2bd23c84-3409-4a96-b587-5d8270a17205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:08.549875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.548072    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-certs" pod "2bd23c84-3409-4a96-b587-5d8270a17205" (UID: "2bd23c84-3409-4a96-b587-5d8270a17205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:08.550429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.550399    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd23c84-3409-4a96-b587-5d8270a17205-istiod-ca-cert" pod "2bd23c84-3409-4a96-b587-5d8270a17205" (UID: "2bd23c84-3409-4a96-b587-5d8270a17205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:08.553981 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.551822    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bd23c84-3409-4a96-b587-5d8270a17205-istio-podinfo" pod "2bd23c84-3409-4a96-b587-5d8270a17205" (UID: "2bd23c84-3409-4a96-b587-5d8270a17205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:21:08.553981 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.552250    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-credential-socket" pod "2bd23c84-3409-4a96-b587-5d8270a17205" (UID: "2bd23c84-3409-4a96-b587-5d8270a17205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:08.553981 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.552433    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-data" pod "2bd23c84-3409-4a96-b587-5d8270a17205" (UID: "2bd23c84-3409-4a96-b587-5d8270a17205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:08.554317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.554295    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-kube-api-access-m4ddg" pod "2bd23c84-3409-4a96-b587-5d8270a17205" (UID: "2bd23c84-3409-4a96-b587-5d8270a17205"). InnerVolumeSpecName "kube-api-access-m4ddg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:08.554482 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.554455    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-envoy" pod "2bd23c84-3409-4a96-b587-5d8270a17205" (UID: "2bd23c84-3409-4a96-b587-5d8270a17205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:08.646307 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.646262    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:08.646307 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.646305    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4ddg\" (UniqueName: \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-kube-api-access-m4ddg\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:08.646524 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.646321    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd23c84-3409-4a96-b587-5d8270a17205-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:08.646524 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.646336    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:08.646524 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.646350    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:08.646524 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.646365    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd23c84-3409-4a96-b587-5d8270a17205-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:08.646524 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.646379    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd23c84-3409-4a96-b587-5d8270a17205-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:08.646524 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:08.646392    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd23c84-3409-4a96-b587-5d8270a17205-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:09.153825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:09.153733    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:21:09.154204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:09.154182    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:21:09.267884 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:09.267833    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 03:21:09.268058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:09.267922    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" podUID="94dba308-852e-4767-a793-5a2e82b13a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 03:21:09.402225 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:09.402194    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"
Oct 30 03:21:09.469559 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:09.469455    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"]
Oct 30 03:21:09.483951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:09.483907    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6qcgn"]
Oct 30 03:21:09.499057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:09.499025    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:10.267277 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.266922    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 03:21:10.267277 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.266981    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" podUID="94dba308-852e-4767-a793-5a2e82b13a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 03:21:10.288576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.288548    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 03:21:10.383649 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.383554    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-data\") pod \"8b8387b9-7f57-4719-935c-1515c8b21ab4\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") "
Oct 30 03:21:10.383649 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.383611    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-token\" (UniqueName: \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-token\") pod \"8b8387b9-7f57-4719-935c-1515c8b21ab4\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") "
Oct 30 03:21:10.383649 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.383637    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-envoy\") pod \"8b8387b9-7f57-4719-935c-1515c8b21ab4\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") "
Oct 30 03:21:10.383986 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.383669    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-podinfo\") pod \"8b8387b9-7f57-4719-935c-1515c8b21ab4\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") "
Oct 30 03:21:10.383986 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.383734    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-credential-socket\") pod \"8b8387b9-7f57-4719-935c-1515c8b21ab4\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") "
Oct 30 03:21:10.383986 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.383795    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-kube-api-access-4g4tm\" (UniqueName: \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-kube-api-access-4g4tm\") pod \"8b8387b9-7f57-4719-935c-1515c8b21ab4\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") "
Oct 30 03:21:10.383986 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.383910    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-socket\") pod \"8b8387b9-7f57-4719-935c-1515c8b21ab4\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") "
Oct 30 03:21:10.383986 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.383955    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8387b9-7f57-4719-935c-1515c8b21ab4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8387b9-7f57-4719-935c-1515c8b21ab4-istiod-ca-cert\") pod \"8b8387b9-7f57-4719-935c-1515c8b21ab4\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") "
Oct 30 03:21:10.384266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.383992    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-certs\") pod \"8b8387b9-7f57-4719-935c-1515c8b21ab4\" (UID: \"8b8387b9-7f57-4719-935c-1515c8b21ab4\") "
Oct 30 03:21:10.384906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.384420    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-data" pod "8b8387b9-7f57-4719-935c-1515c8b21ab4" (UID: "8b8387b9-7f57-4719-935c-1515c8b21ab4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:10.384906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.384647    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-credential-socket" pod "8b8387b9-7f57-4719-935c-1515c8b21ab4" (UID: "8b8387b9-7f57-4719-935c-1515c8b21ab4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:10.384906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.384664    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-certs" pod "8b8387b9-7f57-4719-935c-1515c8b21ab4" (UID: "8b8387b9-7f57-4719-935c-1515c8b21ab4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:10.384906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.384740    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:10.385215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.385021    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8387b9-7f57-4719-935c-1515c8b21ab4-istiod-ca-cert" pod "8b8387b9-7f57-4719-935c-1515c8b21ab4" (UID: "8b8387b9-7f57-4719-935c-1515c8b21ab4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:10.385215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.385086    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-socket" pod "8b8387b9-7f57-4719-935c-1515c8b21ab4" (UID: "8b8387b9-7f57-4719-935c-1515c8b21ab4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:10.386788 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.386763    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-envoy" pod "8b8387b9-7f57-4719-935c-1515c8b21ab4" (UID: "8b8387b9-7f57-4719-935c-1515c8b21ab4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:10.387511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.387485    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-podinfo" pod "8b8387b9-7f57-4719-935c-1515c8b21ab4" (UID: "8b8387b9-7f57-4719-935c-1515c8b21ab4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:21:10.387605 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.387558    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-token" pod "8b8387b9-7f57-4719-935c-1515c8b21ab4" (UID: "8b8387b9-7f57-4719-935c-1515c8b21ab4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:10.387605 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.387592    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-kube-api-access-4g4tm" pod "8b8387b9-7f57-4719-935c-1515c8b21ab4" (UID: "8b8387b9-7f57-4719-935c-1515c8b21ab4"). InnerVolumeSpecName "kube-api-access-4g4tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:10.457873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.454546    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"
Oct 30 03:21:10.457873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.454563    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" event={"ID":"8b8387b9-7f57-4719-935c-1515c8b21ab4","Type":"ContainerDied","Data":"88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e"}
Oct 30 03:21:10.457873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.454637    5061 scope.go:122] "RemoveContainer" containerID="88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e"
Oct 30 03:21:10.457873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.455349    5061 generic.go:356] "Generic (PLEG): container finished" podID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerID="88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e" exitCode=0
Oct 30 03:21:10.457873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.455429    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl" event={"ID":"8b8387b9-7f57-4719-935c-1515c8b21ab4","Type":"ContainerDied","Data":"7904fb97e0a27664ecfe674debc9855a3ddedef539b741521a0826305056a09a"}
Oct 30 03:21:10.467806 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.467782    5061 scope.go:122] "RemoveContainer" containerID="428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22"
Oct 30 03:21:10.478235 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.478215    5061 scope.go:122] "RemoveContainer" containerID="88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e"
Oct 30 03:21:10.478531 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:21:10.478511    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e\": container with ID starting with 88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e not found: ID does not exist" containerID="88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e"
Oct 30 03:21:10.478573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.478541    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e"} err="failed to get container status \"88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e\": rpc error: code = NotFound desc = could not find container \"88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e\": container with ID starting with 88ff7818b82d12207f01291b6b1375ff6d1d23e2024427bfed46a1845c8be27e not found: ID does not exist"
Oct 30 03:21:10.478573 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.478558    5061 scope.go:122] "RemoveContainer" containerID="428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22"
Oct 30 03:21:10.478821 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:21:10.478804    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22\": container with ID starting with 428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22 not found: ID does not exist" containerID="428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22"
Oct 30 03:21:10.478913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.478833    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22"} err="failed to get container status \"428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22\": rpc error: code = NotFound desc = could not find container \"428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22\": container with ID starting with 428cef4e029e3cbec82919ea86988ed6130b6807afc1f328e2c325bfa091cd22 not found: ID does not exist"
Oct 30 03:21:10.486506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.486484    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:10.486506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.486507    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:10.486628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.486515    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8387b9-7f57-4719-935c-1515c8b21ab4-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:10.486628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.486524    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:10.486628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.486535    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4tm\" (UniqueName: \"kubernetes.io/projected/8b8387b9-7f57-4719-935c-1515c8b21ab4-kube-api-access-4g4tm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:10.486628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.486543    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:10.486628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.486551    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8387b9-7f57-4719-935c-1515c8b21ab4-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:10.486628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.486567    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8387b9-7f57-4719-935c-1515c8b21ab4-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:10.505615 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.505585    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"]
Oct 30 03:21:10.510488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.510463    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-9vmrl"]
Oct 30 03:21:10.854436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.854403    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" path="/var/lib/kubelet/pods/2bd23c84-3409-4a96-b587-5d8270a17205/volumes"
Oct 30 03:21:10.854983 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:10.854965    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" path="/var/lib/kubelet/pods/8b8387b9-7f57-4719-935c-1515c8b21ab4/volumes"
Oct 30 03:21:11.151029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:11.150928    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:11.266957 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:11.266917    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 03:21:11.267122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:11.267003    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" podUID="94dba308-852e-4767-a793-5a2e82b13a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 03:21:12.273699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:12.273671    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:12.274179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:12.274161    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:12.275484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:12.275466    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:21:12.982878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:12.982832    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:14.714728 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:14.714694    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:16.259346 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:16.259313    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:16.383242 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:16.383210    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:16.570701 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:16.570670    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:18.208535 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:18.208498    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:19.879941 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:19.879886    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:20.521534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.521501    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"]
Oct 30 03:21:20.521707 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.521689    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn" podUID="2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" containerName="discovery" containerID="cri-o://2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783" gracePeriod=30
Oct 30 03:21:20.807705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.807678    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 03:21:20.840449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.840412    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-token\" (UniqueName: \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-token\") pod \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") "
Oct 30 03:21:20.840699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.840670    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-kube-api-access-4jdtd\" (UniqueName: \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-kube-api-access-4jdtd\") pod \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") "
Oct 30 03:21:20.840776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.840734    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-kubeconfig\") pod \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") "
Oct 30 03:21:20.840876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.840819    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-local-certs\") pod \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") "
Oct 30 03:21:20.840943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.840906    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-ca-configmap\") pod \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") "
Oct 30 03:21:20.841006 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.840971    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-cacerts\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-cacerts\") pod \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") "
Oct 30 03:21:20.841081 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.841051    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-dns-cert\") pod \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\" (UID: \"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb\") "
Oct 30 03:21:20.841834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.841406    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-ca-configmap" pod "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" (UID: "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:20.842003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.841952    5061 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-ca-configmap\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:20.843017 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.842993    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-token" pod "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" (UID: "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:20.843541 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.843514    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-kubeconfig" pod "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" (UID: "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:20.843790 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.843766    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-kube-api-access-4jdtd" pod "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" (UID: "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb"). InnerVolumeSpecName "kube-api-access-4jdtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:20.844030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.844007    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-dns-cert" pod "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" (UID: "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:20.844227 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.844206    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-local-certs" pod "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" (UID: "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:20.844629 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.844607    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-cacerts" pod "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" (UID: "2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:20.943567 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.943534    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdtd\" (UniqueName: \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-kube-api-access-4jdtd\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:20.943567 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.943569    5061 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-kubeconfig\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:20.943988 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.943590    5061 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-local-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:20.943988 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.943605    5061 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-cacerts\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:20.943988 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.943618    5061 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-csr-dns-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:20.943988 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:20.943629    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:21:21.542564 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:21.542532    5061 generic.go:356] "Generic (PLEG): container finished" podID="2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" containerID="2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783" exitCode=0
Oct 30 03:21:21.542726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:21.542581    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"
Oct 30 03:21:21.542726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:21.542613    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn" event={"ID":"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb","Type":"ContainerDied","Data":"2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783"}
Oct 30 03:21:21.542726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:21.542660    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn" event={"ID":"2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb","Type":"ContainerDied","Data":"2fd7c37c50b39b747ff9065d54b5aa5d9327d2e0fe1f55b2e94fc59bbeb4e29c"}
Oct 30 03:21:21.542726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:21.542681    5061 scope.go:122] "RemoveContainer" containerID="2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783"
Oct 30 03:21:21.554314 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:21.554294    5061 scope.go:122] "RemoveContainer" containerID="2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783"
Oct 30 03:21:21.554578 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:21:21.554561    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783\": container with ID starting with 2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783 not found: ID does not exist" containerID="2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783"
Oct 30 03:21:21.554622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:21.554587    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783"} err="failed to get container status \"2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783\": rpc error: code = NotFound desc = could not find container \"2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783\": container with ID starting with 2a3c0485eaa3652c1d6b5270cf19220d1ea53bee021c1abd49e397f04940a783 not found: ID does not exist"
Oct 30 03:21:21.613522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:21.613500    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"]
Oct 30 03:21:21.623873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:21.623839    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5fnmn"]
Oct 30 03:21:21.661391 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:21.661365    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:22.848612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:22.848579    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" path="/var/lib/kubelet/pods/2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb/volumes"
Oct 30 03:21:23.541708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:23.541663    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:25.452540 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:21:25.452503    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:21:25.571403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:25.571360    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:27.629456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:27.629419    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:28.209350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.209318    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"]
Oct 30 03:21:28.210132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210110    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy"
Oct 30 03:21:28.210132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210132    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy"
Oct 30 03:21:28.210337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210163    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" containerName="istio-proxy"
Oct 30 03:21:28.210337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210170    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" containerName="istio-proxy"
Oct 30 03:21:28.210337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210185    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" containerName="discovery"
Oct 30 03:21:28.210337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210192    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" containerName="discovery"
Oct 30 03:21:28.210337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210204    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy"
Oct 30 03:21:28.210337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210212    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy"
Oct 30 03:21:28.210650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210410    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy"
Oct 30 03:21:28.210650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210426    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd23c84-3409-4a96-b587-5d8270a17205" containerName="istio-proxy"
Oct 30 03:21:28.210650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210438    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c5f28eb-0c2b-4f73-a1bb-7977cdb5abbb" containerName="discovery"
Oct 30 03:21:28.210650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.210448    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8387b9-7f57-4719-935c-1515c8b21ab4" containerName="istio-proxy"
Oct 30 03:21:28.216057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.216025    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.231907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.231887    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-ae1dd9\"/\"kube-root-ca.crt\""
Oct 30 03:21:28.242743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.242723    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-ae1dd9\"/\"auth-test-model-ae1dd9-kserve-self-signed-certs\""
Oct 30 03:21:28.242935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.242919    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-ae1dd9\"/\"default-dockercfg-279pp\""
Oct 30 03:21:28.247305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.247285    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"]
Oct 30 03:21:28.290385 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.290362    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:28.334487 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.334458    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-ae1dd9\"/\"openshift-service-ca.crt\""
Oct 30 03:21:28.373289 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.373262    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tmp-dir\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.373421 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.373325    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7mb\" (UniqueName: \"kubernetes.io/projected/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-kube-api-access-pq7mb\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.373540 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.373517    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-home\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.373590 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.373573    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-model-cache\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.373661 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.373639    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tls-certs\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.373815 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.373790    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-dshm\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.476996 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.476898    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7mb\" (UniqueName: \"kubernetes.io/projected/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-kube-api-access-pq7mb\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.477763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.477730    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-home\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.477961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.477797    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-model-cache\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.477961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.477824    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tls-certs\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.478104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.477974    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-dshm\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.478104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.478036    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tmp-dir\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.482902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.478511    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tmp-dir\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.484297 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.484228    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-model-cache\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.500476 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.488423    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-home\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.503610 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.503578    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tls-certs\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.526868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.525275    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7mb\" (UniqueName: \"kubernetes.io/projected/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-kube-api-access-pq7mb\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.557869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.554145    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-dshm\") pod \"auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") " pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:28.829737 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:28.829696    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:29.082281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.082200    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:29.227867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.221661    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"]
Oct 30 03:21:29.246143 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.246067    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:29.580359 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.577105    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"]
Oct 30 03:21:29.590112 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.582696    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:21:29.601542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.599265    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-086aee-dockercfg-r9ws4\""
Oct 30 03:21:29.601542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.600888    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"]
Oct 30 03:21:29.604824 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.604769    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-086aee\""
Oct 30 03:21:29.630306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.626144    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99" event={"ID":"8bf611f0-2273-4646-8cfc-bd12ec8a2dee","Type":"ContainerStarted","Data":"805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485"}
Oct 30 03:21:29.630306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.626202    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99" event={"ID":"8bf611f0-2273-4646-8cfc-bd12ec8a2dee","Type":"ContainerStarted","Data":"a94d0393a80757d4f40c17d77462a5b03a73a0a5f92b9cac56aa03b8fd5dd1e7"}
Oct 30 03:21:29.630306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.626769    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"]
Oct 30 03:21:29.632599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.632576    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:21:29.651595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.651559    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"]
Oct 30 03:21:29.676585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.676513    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99" podStartSLOduration=1.676491543 podStartE2EDuration="1.676491543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:29.67612056 +0000 UTC m=+2434.978298925" watchObservedRunningTime="2026-10-30 03:21:29.676491543 +0000 UTC m=+2434.978669910"
Oct 30 03:21:29.725572 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.725535    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznzj\" (UniqueName: \"kubernetes.io/projected/c12a8d12-5c1d-492c-a468-a65a8f06fadf-kube-api-access-vznzj\") pod \"payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp\" (UID: \"c12a8d12-5c1d-492c-a468-a65a8f06fadf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:21:29.725771 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.725738    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12a8d12-5c1d-492c-a468-a65a8f06fadf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp\" (UID: \"c12a8d12-5c1d-492c-a468-a65a8f06fadf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:21:29.828511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.828460    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccngf\" (UniqueName: \"kubernetes.io/projected/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-kube-api-access-ccngf\") pod \"payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv\" (UID: \"c24ea1dc-460b-4d52-83d5-17c97ca3e29b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:21:29.828724 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.828530    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv\" (UID: \"c24ea1dc-460b-4d52-83d5-17c97ca3e29b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:21:29.829276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.828893    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznzj\" (UniqueName: \"kubernetes.io/projected/c12a8d12-5c1d-492c-a468-a65a8f06fadf-kube-api-access-vznzj\") pod \"payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp\" (UID: \"c12a8d12-5c1d-492c-a468-a65a8f06fadf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:21:29.829276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.829063    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12a8d12-5c1d-492c-a468-a65a8f06fadf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp\" (UID: \"c12a8d12-5c1d-492c-a468-a65a8f06fadf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:21:29.830119 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.830092    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12a8d12-5c1d-492c-a468-a65a8f06fadf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp\" (UID: \"c12a8d12-5c1d-492c-a468-a65a8f06fadf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:21:29.830519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.830173    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:29.872436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.872327    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznzj\" (UniqueName: \"kubernetes.io/projected/c12a8d12-5c1d-492c-a468-a65a8f06fadf-kube-api-access-vznzj\") pod \"payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp\" (UID: \"c12a8d12-5c1d-492c-a468-a65a8f06fadf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:21:29.926518 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.926489    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:21:29.930286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.930253    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccngf\" (UniqueName: \"kubernetes.io/projected/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-kube-api-access-ccngf\") pod \"payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv\" (UID: \"c24ea1dc-460b-4d52-83d5-17c97ca3e29b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:21:29.930778 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.930750    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv\" (UID: \"c24ea1dc-460b-4d52-83d5-17c97ca3e29b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:21:29.932152 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.932128    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv\" (UID: \"c24ea1dc-460b-4d52-83d5-17c97ca3e29b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:21:29.950572 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.950545    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccngf\" (UniqueName: \"kubernetes.io/projected/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-kube-api-access-ccngf\") pod \"payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv\" (UID: \"c24ea1dc-460b-4d52-83d5-17c97ca3e29b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:21:29.950739 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:29.950724    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:21:30.263614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:30.263539    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"]
Oct 30 03:21:30.270061 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:21:30.270034    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12a8d12_5c1d_492c_a468_a65a8f06fadf.slice/crio-93bd0c1ba05134555f629760a0059ba3ad9836eddec94996d32859716533c8b6 WatchSource:0}: Error finding container 93bd0c1ba05134555f629760a0059ba3ad9836eddec94996d32859716533c8b6: Status 404 returned error can't find the container with id 93bd0c1ba05134555f629760a0059ba3ad9836eddec94996d32859716533c8b6
Oct 30 03:21:30.272431 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:30.272414    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:21:30.292239 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:30.292213    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"]
Oct 30 03:21:30.294447 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:21:30.294426    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24ea1dc_460b_4d52_83d5_17c97ca3e29b.slice/crio-7887add4ef09b87b38669a05b2b8412e109a0db3be2be10fb4d5479a6548f911 WatchSource:0}: Error finding container 7887add4ef09b87b38669a05b2b8412e109a0db3be2be10fb4d5479a6548f911: Status 404 returned error can't find the container with id 7887add4ef09b87b38669a05b2b8412e109a0db3be2be10fb4d5479a6548f911
Oct 30 03:21:30.328670 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:30.328631    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:30.633963 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:30.633930    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv" event={"ID":"c24ea1dc-460b-4d52-83d5-17c97ca3e29b","Type":"ContainerStarted","Data":"7887add4ef09b87b38669a05b2b8412e109a0db3be2be10fb4d5479a6548f911"}
Oct 30 03:21:30.635177 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:30.635151    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp" event={"ID":"c12a8d12-5c1d-492c-a468-a65a8f06fadf","Type":"ContainerStarted","Data":"93bd0c1ba05134555f629760a0059ba3ad9836eddec94996d32859716533c8b6"}
Oct 30 03:21:32.020086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:32.020049    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:32.653503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:32.653460    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp" event={"ID":"c12a8d12-5c1d-492c-a468-a65a8f06fadf","Type":"ContainerStarted","Data":"aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206"}
Oct 30 03:21:32.653677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:32.653530    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:21:32.655262 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:32.655236    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv" event={"ID":"c24ea1dc-460b-4d52-83d5-17c97ca3e29b","Type":"ContainerStarted","Data":"aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841"}
Oct 30 03:21:32.655388 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:32.655370    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:21:32.690661 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:32.690620    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp" podStartSLOduration=1.857325074 podStartE2EDuration="3.690608069s" totalImagesPullingTime="1.833282995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:32.689974276 +0000 UTC m=+2437.992152641" watchObservedRunningTime="2026-10-30 03:21:32.690608069 +0000 UTC m=+2437.992786434"
Oct 30 03:21:32.727992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:32.727947    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv" podStartSLOduration=1.931970038 podStartE2EDuration="3.727920479s" totalImagesPullingTime="1.795950441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:32.726928644 +0000 UTC m=+2438.029107013" watchObservedRunningTime="2026-10-30 03:21:32.727920479 +0000 UTC m=+2438.030098847"
Oct 30 03:21:33.857105 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:33.857050    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:35.344624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.344581    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"]
Oct 30 03:21:35.349481 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.349462    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.365483 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.365462    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-734274-serving-cert\""
Oct 30 03:21:35.371689 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.371659    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"]
Oct 30 03:21:35.409799 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.409763    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2slxv\" (UniqueName: \"kubernetes.io/projected/27169abe-153c-4a03-94ed-4c247e23d698-kube-api-access-2slxv\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.410000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.409836    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-maas-api-tls\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.410062 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.410040    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-metrics-certs\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.410153 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.410140    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27169abe-153c-4a03-94ed-4c247e23d698-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.458862 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.458820    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:35.511098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.511066    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-metrics-certs\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.511265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.511181    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27169abe-153c-4a03-94ed-4c247e23d698-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.511331 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.511274    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2slxv\" (UniqueName: \"kubernetes.io/projected/27169abe-153c-4a03-94ed-4c247e23d698-kube-api-access-2slxv\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.511390 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.511329    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-maas-api-tls\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.511475 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:21:35.511452    5061 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-serving-cert: secret "maas-api-e2e-ipp-b-734274-serving-cert" not found
Oct 30 03:21:35.511547 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:21:35.511526    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-maas-api-tls podName:27169abe-153c-4a03-94ed-4c247e23d698 nodeName:}" failed. No retries permitted until 2026-10-30 03:21:36.011505101 +0000 UTC m=+2441.313683457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-maas-api-tls") pod "maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv" (UID: "27169abe-153c-4a03-94ed-4c247e23d698") : secret "maas-api-e2e-ipp-b-734274-serving-cert" not found
Oct 30 03:21:35.511951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.511930    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27169abe-153c-4a03-94ed-4c247e23d698-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.513636 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.513616    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-metrics-certs\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.534529 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.533595    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2slxv\" (UniqueName: \"kubernetes.io/projected/27169abe-153c-4a03-94ed-4c247e23d698-kube-api-access-2slxv\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:35.588585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.588552    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:35.717020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:35.716942    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:36.018513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:36.018435    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-maas-api-tls\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:36.020891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:36.020868    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-maas-api-tls\") pod \"maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:36.264105 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:36.264071    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:36.606868 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:36.599732    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"]
Oct 30 03:21:36.691026 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:36.690988    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv" event={"ID":"27169abe-153c-4a03-94ed-4c247e23d698","Type":"ContainerStarted","Data":"69278038bc9518f13d167edcf4a89faa4a68c8bd8753b8fed200f88a5caca07e"}
Oct 30 03:21:37.499055 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:37.498999    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:37.706478 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:37.706440    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv" event={"ID":"27169abe-153c-4a03-94ed-4c247e23d698","Type":"ContainerStarted","Data":"429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a"}
Oct 30 03:21:37.706947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:37.706544    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:37.754923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:37.754781    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv" podStartSLOduration=2.373480916 podStartE2EDuration="2.754763814s" totalImagesPullingTime="381.282898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:37.746233479 +0000 UTC m=+2443.048411843" watchObservedRunningTime="2026-10-30 03:21:37.754763814 +0000 UTC m=+2443.056942179"
Oct 30 03:21:38.830903 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:38.830868    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:38.849857 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:38.849820    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:39.090349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:39.090273    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:39.739254 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:39.739219    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:21:40.804916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:40.804838    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:42.206422 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:42.206387    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:43.055724 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:43.055689    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:43.289195 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:43.289159    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:43.412762 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:43.412638    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:43.668968 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:43.668890    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:21:43.669116 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:43.668973    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:21:43.720123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:43.720096    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:21:43.828773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:43.828738    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:44.027268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.027191    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"]
Oct 30 03:21:44.033448 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.033427    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.054002 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.053981    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-6b5f19-openshift-default-dockercfg-ks468\""
Oct 30 03:21:44.059186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.059161    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"]
Oct 30 03:21:44.149765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.149731    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e26344-2d16-4f79-8587-9291286d514e-istiod-ca-cert\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.149926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.149782    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-certs\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.149988 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.149954    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-data\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.150092 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.150073    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e26344-2d16-4f79-8587-9291286d514e-istio-podinfo\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.150133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.150121    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-envoy\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.150171 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.150161    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-istio-token\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.150273 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.150250    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-credential-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.150307 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.150286    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.150403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.150391    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm9c5\" (UniqueName: \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-kube-api-access-fm9c5\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252024 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.251987    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-istio-token\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252037    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-credential-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252067    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252171    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm9c5\" (UniqueName: \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-kube-api-access-fm9c5\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252205    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e26344-2d16-4f79-8587-9291286d514e-istiod-ca-cert\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252262    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-certs\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252366    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-data\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252541 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252487    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e26344-2d16-4f79-8587-9291286d514e-istio-podinfo\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252544    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-envoy\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252547    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252574    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-credential-socket\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252832 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252645    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-certs\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252832 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252742    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-data\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.252986 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.252967    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e26344-2d16-4f79-8587-9291286d514e-istiod-ca-cert\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.254905 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.254876    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-envoy\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.255210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.255185    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e26344-2d16-4f79-8587-9291286d514e-istio-podinfo\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.272524 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.272498    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm9c5\" (UniqueName: \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-kube-api-access-fm9c5\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.272896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.272874    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-istio-token\") pod \"e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") " pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.347265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.347228    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:44.671830 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.671804    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"]
Oct 30 03:21:44.674713 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:21:44.674688    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e26344_2d16_4f79_8587_9291286d514e.slice/crio-08f481a9b83aef81ab31e35d637f6fc0d7eaec25f1d9c83f9564d712c1961892 WatchSource:0}: Error finding container 08f481a9b83aef81ab31e35d637f6fc0d7eaec25f1d9c83f9564d712c1961892: Status 404 returned error can't find the container with id 08f481a9b83aef81ab31e35d637f6fc0d7eaec25f1d9c83f9564d712c1961892
Oct 30 03:21:44.677498 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.677461    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:44.677597 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.677533    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:44.677597 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.677573    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:44.766485 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.766451    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" event={"ID":"04e26344-2d16-4f79-8587-9291286d514e","Type":"ContainerStarted","Data":"08903f1fae4401e641a9c6151cc2da3004e2b3da63ce8db11008a61076cf4599"}
Oct 30 03:21:44.766595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.766512    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" event={"ID":"04e26344-2d16-4f79-8587-9291286d514e","Type":"ContainerStarted","Data":"08f481a9b83aef81ab31e35d637f6fc0d7eaec25f1d9c83f9564d712c1961892"}
Oct 30 03:21:44.813021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:44.812958    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" podStartSLOduration=0.812939109 podStartE2EDuration="812.939109ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:44.811781352 +0000 UTC m=+2450.113959716" watchObservedRunningTime="2026-10-30 03:21:44.812939109 +0000 UTC m=+2450.115117475"
Oct 30 03:21:45.348106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:45.348055    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:45.349947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:45.349916    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:21:45.350098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:45.349991    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" podUID="04e26344-2d16-4f79-8587-9291286d514e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 03:21:45.403042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:45.403016    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:46.347741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:46.347701    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:21:46.347963 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:46.347767    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" podUID="04e26344-2d16-4f79-8587-9291286d514e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 03:21:47.339867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:47.339467    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:47.348451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:47.348418    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:21:47.348586 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:47.348468    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" podUID="04e26344-2d16-4f79-8587-9291286d514e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 03:21:48.348449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:48.348410    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:21:48.348972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:48.348477    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" podUID="04e26344-2d16-4f79-8587-9291286d514e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 03:21:48.770480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:48.770381    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:49.043931 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:49.042832    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:49.300387 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:49.300292    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:49.358421 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:49.358383    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:21:49.359013 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:49.358978    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" podUID="04e26344-2d16-4f79-8587-9291286d514e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 03:21:49.758971 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:49.758836    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:49.910220 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:49.910171    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:50.347911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:50.347871    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 03:21:50.348093 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:50.347925    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" podUID="04e26344-2d16-4f79-8587-9291286d514e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 03:21:51.355119 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:51.355084    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:51.452116 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:51.452083    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:51.452584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:51.452564    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:51.453495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:51.453475    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:21:51.495146 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:51.495119    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:51.686688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:51.686611    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:51.927763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:51.927730    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:52.461111 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:52.461082    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:52.625858 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:52.625811    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:53.892759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:53.892724    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:54.146122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.146031    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:54.274694 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.274647    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:54.295641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.295599    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"]
Oct 30 03:21:54.302000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.301972    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:21:54.317129 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.317101    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"]
Oct 30 03:21:54.318716 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.318689    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a0f7e-dockercfg-2gqmn\""
Oct 30 03:21:54.336533 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.336495    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8676ca-1e01-46bd-a45f-45a13be5c888-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk\" (UID: \"3d8676ca-1e01-46bd-a45f-45a13be5c888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:21:54.336533 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.336538    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt7d4\" (UniqueName: \"kubernetes.io/projected/3d8676ca-1e01-46bd-a45f-45a13be5c888-kube-api-access-jt7d4\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk\" (UID: \"3d8676ca-1e01-46bd-a45f-45a13be5c888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:21:54.356115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.356083    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a0f7e\""
Oct 30 03:21:54.469961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.448910    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8676ca-1e01-46bd-a45f-45a13be5c888-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk\" (UID: \"3d8676ca-1e01-46bd-a45f-45a13be5c888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:21:54.469961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.448955    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt7d4\" (UniqueName: \"kubernetes.io/projected/3d8676ca-1e01-46bd-a45f-45a13be5c888-kube-api-access-jt7d4\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk\" (UID: \"3d8676ca-1e01-46bd-a45f-45a13be5c888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:21:54.469961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.451394    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8676ca-1e01-46bd-a45f-45a13be5c888-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk\" (UID: \"3d8676ca-1e01-46bd-a45f-45a13be5c888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:21:54.509730 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.509689    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt7d4\" (UniqueName: \"kubernetes.io/projected/3d8676ca-1e01-46bd-a45f-45a13be5c888-kube-api-access-jt7d4\") pod \"payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk\" (UID: \"3d8676ca-1e01-46bd-a45f-45a13be5c888\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:21:54.618267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.618234    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:21:54.649321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.649284    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:54.917377 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.917334    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:54.920378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:54.920356    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"]
Oct 30 03:21:54.922895 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:21:54.922865    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8676ca_1e01_46bd_a45f_45a13be5c888.slice/crio-cac673c7db7817f86549dd8820486c7ea8d5e0d04be624ea026f6c744e105bf1 WatchSource:0}: Error finding container cac673c7db7817f86549dd8820486c7ea8d5e0d04be624ea026f6c744e105bf1: Status 404 returned error can't find the container with id cac673c7db7817f86549dd8820486c7ea8d5e0d04be624ea026f6c744e105bf1
Oct 30 03:21:55.040766 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:55.040736    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:55.568737 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:21:55.568697    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:21:55.875814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:55.875729    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk" event={"ID":"3d8676ca-1e01-46bd-a45f-45a13be5c888","Type":"ContainerStarted","Data":"d4bfa632ea42f39db38f9b9d6fb18e66020c534fc81c2abf9754730bbdb066cb"}
Oct 30 03:21:55.875814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:55.875793    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk" event={"ID":"3d8676ca-1e01-46bd-a45f-45a13be5c888","Type":"ContainerStarted","Data":"cac673c7db7817f86549dd8820486c7ea8d5e0d04be624ea026f6c744e105bf1"}
Oct 30 03:21:55.876051 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:55.876010    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:21:55.903150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:55.903117    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:55.914542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:55.914491    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk" podStartSLOduration=1.914477398 podStartE2EDuration="1.914477398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:55.910669689 +0000 UTC m=+2461.212848054" watchObservedRunningTime="2026-10-30 03:21:55.914477398 +0000 UTC m=+2461.216655792"
Oct 30 03:21:57.574518 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:57.574481    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:59.184612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:21:59.184572    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:00.923079 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:00.923045    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:02.603612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:02.603576    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"]
Oct 30 03:22:02.638571 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:02.638535    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"]
Oct 30 03:22:02.722221 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:02.722184    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:02.858784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:02.858616    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:03.590871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:03.590821    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:04.067527 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:04.067489    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:04.397165 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:04.397080    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:04.593269 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:04.593238    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:04.746278 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:04.746156    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:06.137822 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:06.137730    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:06.885264 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:06.885230    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:22:07.604494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:07.604454    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4" podUID="87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac" containerName="payload-processing" containerID="cri-o://fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab" gracePeriod=25
Oct 30 03:22:07.639747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:07.639713    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw" podUID="52d98e0b-6b04-4c78-81c0-b0256d9a868a" containerName="payload-pre-processing" containerID="cri-o://bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63" gracePeriod=25
Oct 30 03:22:07.901148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:07.901119    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:22:07.973587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:07.973549    5061 generic.go:356] "Generic (PLEG): container finished" podID="87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac" containerID="fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab" exitCode=0
Oct 30 03:22:07.973763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:07.973590    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"
Oct 30 03:22:07.973763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:07.973628    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4" event={"ID":"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac","Type":"ContainerDied","Data":"fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab"}
Oct 30 03:22:07.973763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:07.973709    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4" event={"ID":"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac","Type":"ContainerDied","Data":"3275877b33b30103ecec22c44ce9b17d676224d1529318cf892d9552945f958a"}
Oct 30 03:22:07.973763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:07.973736    5061 scope.go:122] "RemoveContainer" containerID="fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab"
Oct 30 03:22:07.987119 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:07.987090    5061 scope.go:122] "RemoveContainer" containerID="fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab"
Oct 30 03:22:07.987443 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:22:07.987421    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab\": container with ID starting with fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab not found: ID does not exist" containerID="fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab"
Oct 30 03:22:07.987541 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:07.987455    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab"} err="failed to get container status \"fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab\": rpc error: code = NotFound desc = could not find container \"fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab\": container with ID starting with fc0a4e87551298a658c1dac4e620d7fead2906b1d405ef6e335a490a997e35ab not found: ID does not exist"
Oct 30 03:22:08.069637 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.069609    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-kube-api-access-b4qzb\" (UniqueName: \"kubernetes.io/projected/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-kube-api-access-b4qzb\") pod \"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac\" (UID: \"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac\") "
Oct 30 03:22:08.069775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.069741    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-plugins-config-volume\") pod \"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac\" (UID: \"87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac\") "
Oct 30 03:22:08.070140 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.070114    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-plugins-config-volume" pod "87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac" (UID: "87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:08.070374 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.070353    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:22:08.072115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.072092    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-kube-api-access-b4qzb" pod "87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac" (UID: "87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac"). InnerVolumeSpecName "kube-api-access-b4qzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:08.154947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.154887    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:08.171818 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.171787    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4qzb\" (UniqueName: \"kubernetes.io/projected/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac-kube-api-access-b4qzb\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:22:08.317906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.317878    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"]
Oct 30 03:22:08.328365 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.328335    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cf8f5-68468b9994-vb6n4"]
Oct 30 03:22:08.766345 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.766314    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:08.850420 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.850388    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac" path="/var/lib/kubelet/pods/87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac/volumes"
Oct 30 03:22:08.928056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:08.928024    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:10.069139 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:10.069091    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:10.517098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:10.517019    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:10.667817 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:10.667787    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:12.388057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:12.388020    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw" podUID="52d98e0b-6b04-4c78-81c0-b0256d9a868a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:22:12.725557 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:12.725434    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:15.067024 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:15.066970    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:16.895427 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:16.895370    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:18.939817 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:18.939772    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:20.688529 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:20.688445    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:21.606232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:21.606199    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:22.026321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:22.026238    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"]
Oct 30 03:22:22.091117 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:22.091083    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"]
Oct 30 03:22:22.263098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:22.263065    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:22.388065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:22.388027    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw" podUID="52d98e0b-6b04-4c78-81c0-b0256d9a868a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:22:22.408234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:22.408200    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:22.593904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:22.593841    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:24.710281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:24.710241    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:25.449038 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:22:25.449003    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:22:26.710609 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:26.710558    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:27.027174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.027087    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5" podUID="1805274d-4b09-4dad-ade2-1e42d4caf20f" containerName="payload-processing" containerID="cri-o://9e7a5eb08ca9bc43855a26ea412785d55e85fbb898ee5ecbeae6deffdce825dd" gracePeriod=25
Oct 30 03:22:27.091933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.091898    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9" podUID="e8af7a5c-9090-4033-b2d6-4acaa05490c6" containerName="payload-pre-processing" containerID="cri-o://806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471" gracePeriod=25
Oct 30 03:22:27.149065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.149039    5061 generic.go:356] "Generic (PLEG): container finished" podID="1805274d-4b09-4dad-ade2-1e42d4caf20f" containerID="9e7a5eb08ca9bc43855a26ea412785d55e85fbb898ee5ecbeae6deffdce825dd" exitCode=0
Oct 30 03:22:27.149209 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.149118    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5" event={"ID":"1805274d-4b09-4dad-ade2-1e42d4caf20f","Type":"ContainerDied","Data":"9e7a5eb08ca9bc43855a26ea412785d55e85fbb898ee5ecbeae6deffdce825dd"}
Oct 30 03:22:27.397323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.397294    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:22:27.439091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.439068    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:22:27.477767 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.477739    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1805274d-4b09-4dad-ade2-1e42d4caf20f-kube-api-access-gmclt\" (UniqueName: \"kubernetes.io/projected/1805274d-4b09-4dad-ade2-1e42d4caf20f-kube-api-access-gmclt\") pod \"1805274d-4b09-4dad-ade2-1e42d4caf20f\" (UID: \"1805274d-4b09-4dad-ade2-1e42d4caf20f\") "
Oct 30 03:22:27.478018 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.477994    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1805274d-4b09-4dad-ade2-1e42d4caf20f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1805274d-4b09-4dad-ade2-1e42d4caf20f-plugins-config-volume\") pod \"1805274d-4b09-4dad-ade2-1e42d4caf20f\" (UID: \"1805274d-4b09-4dad-ade2-1e42d4caf20f\") "
Oct 30 03:22:27.478432 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.478400    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1805274d-4b09-4dad-ade2-1e42d4caf20f-plugins-config-volume" pod "1805274d-4b09-4dad-ade2-1e42d4caf20f" (UID: "1805274d-4b09-4dad-ade2-1e42d4caf20f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:27.478663 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.478640    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1805274d-4b09-4dad-ade2-1e42d4caf20f-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:22:27.480550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.480526    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1805274d-4b09-4dad-ade2-1e42d4caf20f-kube-api-access-gmclt" pod "1805274d-4b09-4dad-ade2-1e42d4caf20f" (UID: "1805274d-4b09-4dad-ade2-1e42d4caf20f"). InnerVolumeSpecName "kube-api-access-gmclt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:27.580175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.580098    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8af7a5c-9090-4033-b2d6-4acaa05490c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8af7a5c-9090-4033-b2d6-4acaa05490c6-plugins-config-volume\") pod \"e8af7a5c-9090-4033-b2d6-4acaa05490c6\" (UID: \"e8af7a5c-9090-4033-b2d6-4acaa05490c6\") "
Oct 30 03:22:27.580175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.580148    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8af7a5c-9090-4033-b2d6-4acaa05490c6-kube-api-access-z46f4\" (UniqueName: \"kubernetes.io/projected/e8af7a5c-9090-4033-b2d6-4acaa05490c6-kube-api-access-z46f4\") pod \"e8af7a5c-9090-4033-b2d6-4acaa05490c6\" (UID: \"e8af7a5c-9090-4033-b2d6-4acaa05490c6\") "
Oct 30 03:22:27.580555 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.580530    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8af7a5c-9090-4033-b2d6-4acaa05490c6-plugins-config-volume" pod "e8af7a5c-9090-4033-b2d6-4acaa05490c6" (UID: "e8af7a5c-9090-4033-b2d6-4acaa05490c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:27.580913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.580889    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmclt\" (UniqueName: \"kubernetes.io/projected/1805274d-4b09-4dad-ade2-1e42d4caf20f-kube-api-access-gmclt\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:22:27.580913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.580908    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8af7a5c-9090-4033-b2d6-4acaa05490c6-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:22:27.582470 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.582448    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8af7a5c-9090-4033-b2d6-4acaa05490c6-kube-api-access-z46f4" pod "e8af7a5c-9090-4033-b2d6-4acaa05490c6" (UID: "e8af7a5c-9090-4033-b2d6-4acaa05490c6"). InnerVolumeSpecName "kube-api-access-z46f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:27.682224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.682200    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z46f4\" (UniqueName: \"kubernetes.io/projected/e8af7a5c-9090-4033-b2d6-4acaa05490c6-kube-api-access-z46f4\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:22:27.911999 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:27.911974    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:22:28.089085 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.089054    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d98e0b-6b04-4c78-81c0-b0256d9a868a-kube-api-access-fssjt\" (UniqueName: \"kubernetes.io/projected/52d98e0b-6b04-4c78-81c0-b0256d9a868a-kube-api-access-fssjt\") pod \"52d98e0b-6b04-4c78-81c0-b0256d9a868a\" (UID: \"52d98e0b-6b04-4c78-81c0-b0256d9a868a\") "
Oct 30 03:22:28.089468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.089446    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d98e0b-6b04-4c78-81c0-b0256d9a868a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d98e0b-6b04-4c78-81c0-b0256d9a868a-plugins-config-volume\") pod \"52d98e0b-6b04-4c78-81c0-b0256d9a868a\" (UID: \"52d98e0b-6b04-4c78-81c0-b0256d9a868a\") "
Oct 30 03:22:28.089807 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.089783    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d98e0b-6b04-4c78-81c0-b0256d9a868a-plugins-config-volume" pod "52d98e0b-6b04-4c78-81c0-b0256d9a868a" (UID: "52d98e0b-6b04-4c78-81c0-b0256d9a868a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:28.089976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.089960    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d98e0b-6b04-4c78-81c0-b0256d9a868a-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:22:28.091275 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.091243    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d98e0b-6b04-4c78-81c0-b0256d9a868a-kube-api-access-fssjt" pod "52d98e0b-6b04-4c78-81c0-b0256d9a868a" (UID: "52d98e0b-6b04-4c78-81c0-b0256d9a868a"). InnerVolumeSpecName "kube-api-access-fssjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:28.157388 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.157315    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"
Oct 30 03:22:28.157525 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.157314    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5" event={"ID":"1805274d-4b09-4dad-ade2-1e42d4caf20f","Type":"ContainerDied","Data":"427e35a14c9bfce53e8342da219fd1e9397331afcb12c16283f81a8ed368eda3"}
Oct 30 03:22:28.157525 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.157450    5061 scope.go:122] "RemoveContainer" containerID="9e7a5eb08ca9bc43855a26ea412785d55e85fbb898ee5ecbeae6deffdce825dd"
Oct 30 03:22:28.158769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.158736    5061 generic.go:356] "Generic (PLEG): container finished" podID="e8af7a5c-9090-4033-b2d6-4acaa05490c6" containerID="806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471" exitCode=0
Oct 30 03:22:28.158879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.158784    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"
Oct 30 03:22:28.158879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.158819    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9" event={"ID":"e8af7a5c-9090-4033-b2d6-4acaa05490c6","Type":"ContainerDied","Data":"806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471"}
Oct 30 03:22:28.158988 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.158884    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9" event={"ID":"e8af7a5c-9090-4033-b2d6-4acaa05490c6","Type":"ContainerDied","Data":"8ea97f3362f0d5db52410cdcb030d96b19134b302349e054fb86f79d29e408f6"}
Oct 30 03:22:28.160284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.160262    5061 generic.go:356] "Generic (PLEG): container finished" podID="52d98e0b-6b04-4c78-81c0-b0256d9a868a" containerID="bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63" exitCode=0
Oct 30 03:22:28.160389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.160301    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"
Oct 30 03:22:28.160389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.160342    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw" event={"ID":"52d98e0b-6b04-4c78-81c0-b0256d9a868a","Type":"ContainerDied","Data":"bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63"}
Oct 30 03:22:28.160389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.160383    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw" event={"ID":"52d98e0b-6b04-4c78-81c0-b0256d9a868a","Type":"ContainerDied","Data":"336230331e4672e8f8d62f0e34588ca2f6d83bfe84954426bbba7db14ec5dcaa"}
Oct 30 03:22:28.170413 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.170389    5061 scope.go:122] "RemoveContainer" containerID="806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471"
Oct 30 03:22:28.180248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.180232    5061 scope.go:122] "RemoveContainer" containerID="806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471"
Oct 30 03:22:28.180499 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:22:28.180480    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471\": container with ID starting with 806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471 not found: ID does not exist" containerID="806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471"
Oct 30 03:22:28.180547 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.180507    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471"} err="failed to get container status \"806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471\": rpc error: code = NotFound desc = could not find container \"806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471\": container with ID starting with 806530f64b8e9f40a94fec6d5a665269a3a41cc5205b20bc88391746008d3471 not found: ID does not exist"
Oct 30 03:22:28.180547 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.180522    5061 scope.go:122] "RemoveContainer" containerID="bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63"
Oct 30 03:22:28.189695 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.189672    5061 scope.go:122] "RemoveContainer" containerID="bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63"
Oct 30 03:22:28.190034 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:22:28.190007    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63\": container with ID starting with bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63 not found: ID does not exist" containerID="bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63"
Oct 30 03:22:28.190128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.190047    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63"} err="failed to get container status \"bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63\": rpc error: code = NotFound desc = could not find container \"bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63\": container with ID starting with bebec751c90379a4d54de5811c9b4177485212b6f1d8af496ffa0a94ec8b3d63 not found: ID does not exist"
Oct 30 03:22:28.190904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.190824    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fssjt\" (UniqueName: \"kubernetes.io/projected/52d98e0b-6b04-4c78-81c0-b0256d9a868a-kube-api-access-fssjt\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:22:28.230842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.230815    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"]
Oct 30 03:22:28.235728 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.235709    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-648f7f965f-z5qm9"]
Oct 30 03:22:28.265912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.265881    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"]
Oct 30 03:22:28.271725 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.271698    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-659ddbf7cb-6qxs5"]
Oct 30 03:22:28.301105 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.301078    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"]
Oct 30 03:22:28.305657 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.305631    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cf8f5-685558f87-f6gbw"]
Oct 30 03:22:28.415038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.414961    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:28.850518 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.850489    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1805274d-4b09-4dad-ade2-1e42d4caf20f" path="/var/lib/kubelet/pods/1805274d-4b09-4dad-ade2-1e42d4caf20f/volumes"
Oct 30 03:22:28.850961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.850945    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d98e0b-6b04-4c78-81c0-b0256d9a868a" path="/var/lib/kubelet/pods/52d98e0b-6b04-4c78-81c0-b0256d9a868a/volumes"
Oct 30 03:22:28.851346 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:28.851333    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8af7a5c-9090-4033-b2d6-4acaa05490c6" path="/var/lib/kubelet/pods/e8af7a5c-9090-4033-b2d6-4acaa05490c6/volumes"
Oct 30 03:22:30.198459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:30.198422    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:31.426787 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.426751    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:31.685130 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.685049    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:31.994976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.994897    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"]
Oct 30 03:22:31.995795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.995740    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1805274d-4b09-4dad-ade2-1e42d4caf20f" containerName="payload-processing"
Oct 30 03:22:31.995795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.995765    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1805274d-4b09-4dad-ade2-1e42d4caf20f" containerName="payload-processing"
Oct 30 03:22:31.995795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.995787    5061 cpu_manager.go:414] "containerMap: removing container" podUID="87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac" containerName="payload-processing"
Oct 30 03:22:31.995795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.995794    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac" containerName="payload-processing"
Oct 30 03:22:31.996132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.995830    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e8af7a5c-9090-4033-b2d6-4acaa05490c6" containerName="payload-pre-processing"
Oct 30 03:22:31.996132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.995839    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8af7a5c-9090-4033-b2d6-4acaa05490c6" containerName="payload-pre-processing"
Oct 30 03:22:31.996132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.995886    5061 cpu_manager.go:414] "containerMap: removing container" podUID="52d98e0b-6b04-4c78-81c0-b0256d9a868a" containerName="payload-pre-processing"
Oct 30 03:22:31.996132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.995893    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d98e0b-6b04-4c78-81c0-b0256d9a868a" containerName="payload-pre-processing"
Oct 30 03:22:31.996132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.996088    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8af7a5c-9090-4033-b2d6-4acaa05490c6" containerName="payload-pre-processing"
Oct 30 03:22:31.996132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.996105    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d98e0b-6b04-4c78-81c0-b0256d9a868a" containerName="payload-pre-processing"
Oct 30 03:22:31.996132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.996129    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1805274d-4b09-4dad-ade2-1e42d4caf20f" containerName="payload-processing"
Oct 30 03:22:31.996376 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:31.996139    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ae0eb6-6e7a-4cc9-8358-d8ff2a6fbcac" containerName="payload-processing"
Oct 30 03:22:32.001861 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.001831    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:22:32.026265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.026230    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-146bb7\""
Oct 30 03:22:32.032591 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.032558    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"]
Oct 30 03:22:32.076869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.068928    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83428494-94f2-427a-a101-2cc2d4f45a24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4\" (UID: \"83428494-94f2-427a-a101-2cc2d4f45a24\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:22:32.076869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.069020    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrzvc\" (UniqueName: \"kubernetes.io/projected/83428494-94f2-427a-a101-2cc2d4f45a24-kube-api-access-zrzvc\") pod \"payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4\" (UID: \"83428494-94f2-427a-a101-2cc2d4f45a24\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:22:32.164281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.164248    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"]
Oct 30 03:22:32.168766 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.168748    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:22:32.170133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.170107    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83428494-94f2-427a-a101-2cc2d4f45a24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4\" (UID: \"83428494-94f2-427a-a101-2cc2d4f45a24\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:22:32.170251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.170201    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrzvc\" (UniqueName: \"kubernetes.io/projected/83428494-94f2-427a-a101-2cc2d4f45a24-kube-api-access-zrzvc\") pod \"payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4\" (UID: \"83428494-94f2-427a-a101-2cc2d4f45a24\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:22:32.170833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.170813    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83428494-94f2-427a-a101-2cc2d4f45a24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4\" (UID: \"83428494-94f2-427a-a101-2cc2d4f45a24\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:22:32.182976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.182954    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-146bb7-dockercfg-7vgx4\""
Oct 30 03:22:32.187899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.187876    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"]
Oct 30 03:22:32.191525 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.191501    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrzvc\" (UniqueName: \"kubernetes.io/projected/83428494-94f2-427a-a101-2cc2d4f45a24-kube-api-access-zrzvc\") pod \"payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4\" (UID: \"83428494-94f2-427a-a101-2cc2d4f45a24\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:22:32.271275 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.271194    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae61a65-0a49-4b8b-935f-45bb6981e8ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952\" (UID: \"aae61a65-0a49-4b8b-935f-45bb6981e8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:22:32.271458 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.271440    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtw7s\" (UniqueName: \"kubernetes.io/projected/aae61a65-0a49-4b8b-935f-45bb6981e8ba-kube-api-access-vtw7s\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952\" (UID: \"aae61a65-0a49-4b8b-935f-45bb6981e8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:22:32.316979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.316954    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:22:32.374270 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.372731    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtw7s\" (UniqueName: \"kubernetes.io/projected/aae61a65-0a49-4b8b-935f-45bb6981e8ba-kube-api-access-vtw7s\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952\" (UID: \"aae61a65-0a49-4b8b-935f-45bb6981e8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:22:32.374270 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.372916    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae61a65-0a49-4b8b-935f-45bb6981e8ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952\" (UID: \"aae61a65-0a49-4b8b-935f-45bb6981e8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:22:32.374270 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.373966    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae61a65-0a49-4b8b-935f-45bb6981e8ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952\" (UID: \"aae61a65-0a49-4b8b-935f-45bb6981e8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:22:32.399155 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.399128    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtw7s\" (UniqueName: \"kubernetes.io/projected/aae61a65-0a49-4b8b-935f-45bb6981e8ba-kube-api-access-vtw7s\") pod \"payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952\" (UID: \"aae61a65-0a49-4b8b-935f-45bb6981e8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:22:32.483025 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.482987    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:22:32.628503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.628467    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"]
Oct 30 03:22:32.633967 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:22:32.633937    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83428494_94f2_427a_a101_2cc2d4f45a24.slice/crio-c268e2076710f143d616a9c52ca32e955f3d0020b41ef9a83771d6079e50c921 WatchSource:0}: Error finding container c268e2076710f143d616a9c52ca32e955f3d0020b41ef9a83771d6079e50c921: Status 404 returned error can't find the container with id c268e2076710f143d616a9c52ca32e955f3d0020b41ef9a83771d6079e50c921
Oct 30 03:22:32.786915 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:32.786888    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"]
Oct 30 03:22:32.787790 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:22:32.787758    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae61a65_0a49_4b8b_935f_45bb6981e8ba.slice/crio-f7b97ed5c2e63376343b138179d1088383f2ffd9d3bfda2079ee5db6a8a7eda6 WatchSource:0}: Error finding container f7b97ed5c2e63376343b138179d1088383f2ffd9d3bfda2079ee5db6a8a7eda6: Status 404 returned error can't find the container with id f7b97ed5c2e63376343b138179d1088383f2ffd9d3bfda2079ee5db6a8a7eda6
Oct 30 03:22:33.142226 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:33.142194    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:33.205879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:33.205805    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952" event={"ID":"aae61a65-0a49-4b8b-935f-45bb6981e8ba","Type":"ContainerStarted","Data":"2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5"}
Oct 30 03:22:33.206062 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:33.205943    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952" event={"ID":"aae61a65-0a49-4b8b-935f-45bb6981e8ba","Type":"ContainerStarted","Data":"f7b97ed5c2e63376343b138179d1088383f2ffd9d3bfda2079ee5db6a8a7eda6"}
Oct 30 03:22:33.208034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:33.207131    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:22:33.209287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:33.209253    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4" event={"ID":"83428494-94f2-427a-a101-2cc2d4f45a24","Type":"ContainerStarted","Data":"e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56"}
Oct 30 03:22:33.209412 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:33.209312    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4" event={"ID":"83428494-94f2-427a-a101-2cc2d4f45a24","Type":"ContainerStarted","Data":"c268e2076710f143d616a9c52ca32e955f3d0020b41ef9a83771d6079e50c921"}
Oct 30 03:22:33.210014 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:33.209991    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:22:33.247229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:33.247179    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952" podStartSLOduration=1.247166056 podStartE2EDuration="1.247166056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:33.244927395 +0000 UTC m=+2498.547105762" watchObservedRunningTime="2026-10-30 03:22:33.247166056 +0000 UTC m=+2498.549344418"
Oct 30 03:22:33.288984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:33.288937    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4" podStartSLOduration=2.288923404 podStartE2EDuration="2.288923404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:33.286131818 +0000 UTC m=+2498.588310183" watchObservedRunningTime="2026-10-30 03:22:33.288923404 +0000 UTC m=+2498.591101769"
Oct 30 03:22:35.244360 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.244328    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"]
Oct 30 03:22:35.249035 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.249014    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.277218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.277190    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"]
Oct 30 03:22:35.312148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.312117    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.312148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.312155    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.312379 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.312175    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.312459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.312386    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q52hb\" (UniqueName: \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-kube-api-access-q52hb\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.312523 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.312484    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.312523 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.312511    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.312651 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.312620    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.312778 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.312760    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a074e7fd-bbf0-4235-ac85-82f9081ec762-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.312883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.312800    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.340431 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.340404    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"]
Oct 30 03:22:35.344994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.344977    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.359055 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.359035    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1a0f7e-serving-cert\""
Oct 30 03:22:35.363682 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.363664    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"]
Oct 30 03:22:35.415539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.415500    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a074e7fd-bbf0-4235-ac85-82f9081ec762-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.415698 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.415553    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.415742 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.415701    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.415742 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.415734    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.415840 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.415764    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.415840 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.415832    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.415964 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.415910    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q52hb\" (UniqueName: \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-kube-api-access-q52hb\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.416068 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.415990    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.416068 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.416031    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.416206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.416073    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.416206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.416117    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.416206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.416142    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0166ec2f-c5c8-4387-91a7-15093053f4c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.416206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.416199    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk55m\" (UniqueName: \"kubernetes.io/projected/0166ec2f-c5c8-4387-91a7-15093053f4c4-kube-api-access-vk55m\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.416416 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.416282    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.416827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.416806    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a074e7fd-bbf0-4235-ac85-82f9081ec762-istiod-ca-cert\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.417055 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.417037    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-credential-socket\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.423913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.419874    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-certs\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.423913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.420122    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-data\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.423913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.422422    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-podinfo\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.435437 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.424377    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-envoy\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.438461 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.438441    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-token\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.440643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.440620    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q52hb\" (UniqueName: \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-kube-api-access-q52hb\") pod \"e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") " pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.457178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.457144    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:35.517628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.517549    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk55m\" (UniqueName: \"kubernetes.io/projected/0166ec2f-c5c8-4387-91a7-15093053f4c4-kube-api-access-vk55m\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.517751 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.517722    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.517812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.517802    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.517898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.517835    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0166ec2f-c5c8-4387-91a7-15093053f4c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.518501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.518477    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0166ec2f-c5c8-4387-91a7-15093053f4c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.520365 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.520338    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.520450 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.520343    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.534969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.534943    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk55m\" (UniqueName: \"kubernetes.io/projected/0166ec2f-c5c8-4387-91a7-15093053f4c4-kube-api-access-vk55m\") pod \"maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.566929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.566895    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:35.586663 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.586190    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:35.659142 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.659116    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:35.924804 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.924769    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"]
Oct 30 03:22:35.927830 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:22:35.927791    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda074e7fd_bbf0_4235_ac85_82f9081ec762.slice/crio-7a07081f5041ba29ec9473d76f90ae2fc7703af6c0d5d49aaebe4d4537434ef3 WatchSource:0}: Error finding container 7a07081f5041ba29ec9473d76f90ae2fc7703af6c0d5d49aaebe4d4537434ef3: Status 404 returned error can't find the container with id 7a07081f5041ba29ec9473d76f90ae2fc7703af6c0d5d49aaebe4d4537434ef3
Oct 30 03:22:35.930894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.930863    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:35.930976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.930927    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:35.930976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:35.930966    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:36.001580 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:36.001552    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"]
Oct 30 03:22:36.004151 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:22:36.004124    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0166ec2f_c5c8_4387_91a7_15093053f4c4.slice/crio-4bfd6f8239f64e862983900efd5a7429ab8e8576efcd8f6c1dc35241921dbcb4 WatchSource:0}: Error finding container 4bfd6f8239f64e862983900efd5a7429ab8e8576efcd8f6c1dc35241921dbcb4: Status 404 returned error can't find the container with id 4bfd6f8239f64e862983900efd5a7429ab8e8576efcd8f6c1dc35241921dbcb4
Oct 30 03:22:36.234943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:36.231827    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h" event={"ID":"0166ec2f-c5c8-4387-91a7-15093053f4c4","Type":"ContainerStarted","Data":"4bfd6f8239f64e862983900efd5a7429ab8e8576efcd8f6c1dc35241921dbcb4"}
Oct 30 03:22:36.234943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:36.233530    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk" event={"ID":"a074e7fd-bbf0-4235-ac85-82f9081ec762","Type":"ContainerStarted","Data":"807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b"}
Oct 30 03:22:36.234943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:36.233567    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk" event={"ID":"a074e7fd-bbf0-4235-ac85-82f9081ec762","Type":"ContainerStarted","Data":"7a07081f5041ba29ec9473d76f90ae2fc7703af6c0d5d49aaebe4d4537434ef3"}
Oct 30 03:22:36.286366 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:36.286300    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk" podStartSLOduration=1.2862796859999999 podStartE2EDuration="1.286279686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:36.276495619 +0000 UTC m=+2501.578673986" watchObservedRunningTime="2026-10-30 03:22:36.286279686 +0000 UTC m=+2501.588458055"
Oct 30 03:22:36.567865 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:36.567803    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:37.077623 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:37.077577    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:37.258894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:37.255802    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h" event={"ID":"0166ec2f-c5c8-4387-91a7-15093053f4c4","Type":"ContainerStarted","Data":"a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983"}
Oct 30 03:22:37.258894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:37.256220    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:37.298934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:37.298863    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h" podStartSLOduration=1.903671981 podStartE2EDuration="2.29882951s" totalImagesPullingTime="395.157529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:37.297289332 +0000 UTC m=+2502.599467701" watchObservedRunningTime="2026-10-30 03:22:37.29882951 +0000 UTC m=+2502.601007875"
Oct 30 03:22:37.568974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:37.568938    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:22:37.569199 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:37.569039    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk" podUID="a074e7fd-bbf0-4235-ac85-82f9081ec762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 03:22:37.796161 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:37.796130    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:38.077291 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:38.077263    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:38.345299 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:38.345187    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:38.568569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:38.568524    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:22:38.568747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:38.568587    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk" podUID="a074e7fd-bbf0-4235-ac85-82f9081ec762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 03:22:38.666153 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:38.666073    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:39.568009 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:39.567973    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:22:39.568494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:39.568031    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk" podUID="a074e7fd-bbf0-4235-ac85-82f9081ec762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 03:22:40.052755 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.052727    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"]
Oct 30 03:22:40.058417 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.058398    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.071686 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.071661    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-ae1dd9-openshift-default-dockercfg-bvl72\""
Oct 30 03:22:40.087409 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.087384    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"]
Oct 30 03:22:40.111468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.111440    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-envoy\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.111604 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.111584    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-data\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.111672 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.111642    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-token\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.111732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.111675    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-certs\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.111842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.111817    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-credential-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.112013 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.111994    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-podinfo\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.112087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.112028    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.112087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.112068    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnh8g\" (UniqueName: \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-kube-api-access-tnh8g\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.112201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.112184    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istiod-ca-cert\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214088    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-envoy\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214289 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214183    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-data\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214289 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214220    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-token\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214289 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214242    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-certs\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214308    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-credential-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214483 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214415    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-podinfo\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214483 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214446    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214600 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214481    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnh8g\" (UniqueName: \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-kube-api-access-tnh8g\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214659 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214597    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istiod-ca-cert\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214659 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214615    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-data\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214664    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-certs\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214742    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-credential-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.214880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.214836    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-socket\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.215188 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.215166    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istiod-ca-cert\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.216738 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.216711    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-envoy\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.217026 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.217005    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-podinfo\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.248872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.246562    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-token\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.249080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.249034    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnh8g\" (UniqueName: \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-kube-api-access-tnh8g\") pod \"e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") " pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.372650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.372564    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:40.546754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.546487    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:40.568236 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.568196    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:22:40.568648 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.568265    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk" podUID="a074e7fd-bbf0-4235-ac85-82f9081ec762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 03:22:40.748107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.748075    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"]
Oct 30 03:22:40.750038 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:22:40.750008    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ce09f3_daae_4d0f_ae5c_41dd7f88fc2f.slice/crio-8ee9161a6c4ccfa81462b2de498292690165ac8a0a273d872350aacbee8d9755 WatchSource:0}: Error finding container 8ee9161a6c4ccfa81462b2de498292690165ac8a0a273d872350aacbee8d9755: Status 404 returned error can't find the container with id 8ee9161a6c4ccfa81462b2de498292690165ac8a0a273d872350aacbee8d9755
Oct 30 03:22:40.753267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.753225    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:40.753389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.753307    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:40.753389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:40.753354    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:41.287916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:41.287880    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" event={"ID":"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f","Type":"ContainerStarted","Data":"c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426"}
Oct 30 03:22:41.288099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:41.287943    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" event={"ID":"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f","Type":"ContainerStarted","Data":"8ee9161a6c4ccfa81462b2de498292690165ac8a0a273d872350aacbee8d9755"}
Oct 30 03:22:41.347163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:41.347095    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" podStartSLOduration=1.347075522 podStartE2EDuration="1.347075522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:41.336319426 +0000 UTC m=+2506.638497790" watchObservedRunningTime="2026-10-30 03:22:41.347075522 +0000 UTC m=+2506.649253888"
Oct 30 03:22:41.373316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:41.373268    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:41.380907 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:41.380823    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 03:22:41.381142 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:41.380937    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 03:22:41.568871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:41.568336    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:22:41.568871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:41.568418    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk" podUID="a074e7fd-bbf0-4235-ac85-82f9081ec762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 03:22:41.700557 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:41.700522    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:42.312816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:42.312773    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:42.373095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:42.373053    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 03:22:42.373266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:42.373136    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 03:22:42.668836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:42.668752    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:42.669342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:42.669203    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:42.670117 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:42.670094    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:22:43.269230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:43.269200    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:22:43.373734 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:43.373695    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 03:22:43.373951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:43.373800    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 03:22:43.987158 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:43.987128    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:44.217859 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:44.217824    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:22:44.373270 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:44.373237    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 03:22:44.373448 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:44.373289    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 03:22:45.240871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:45.226977    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:22:45.373837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:45.373800    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 03:22:45.374041 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:45.373882    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 03:22:45.575813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:45.575775    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:45.742416 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:45.742384    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:45.944493 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:45.944374    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:46.166872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:46.155961    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:46.373926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:46.373893    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 03:22:46.374333 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:46.373944    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 03:22:46.670114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:46.670031    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:47.360935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:47.357521    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:47.389447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:47.389410    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 03:22:47.490668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:47.490617    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 03:22:47.541040 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:47.541008    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:47.859163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:47.859129    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:48.195131 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:48.195037    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:48.377535 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:48.377506    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:48.378021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:48.378001    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:48.378628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:48.378611    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:22:49.701908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:49.701868    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:49.854179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:49.854133    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:51.211783 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:51.211734    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:51.372294 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:51.372263    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:51.589367 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:51.589333    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:52.374765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:52.374729    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:52.593350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:52.593318    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:53.242048 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.241997    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:53.389344 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.389303    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:53.459025 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.458991    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"]
Oct 30 03:22:53.464278 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.464247    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:22:53.524983 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.514530    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-4b5459-dockercfg-ptpxw\""
Oct 30 03:22:53.525576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.519733    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-4b5459\""
Oct 30 03:22:53.532933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.530397    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"]
Oct 30 03:22:53.560326 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.560275    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhcxv\" (UniqueName: \"kubernetes.io/projected/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-kube-api-access-vhcxv\") pod \"payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5\" (UID: \"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:22:53.560879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.560836    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5\" (UID: \"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:22:53.662165 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.662133    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhcxv\" (UniqueName: \"kubernetes.io/projected/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-kube-api-access-vhcxv\") pod \"payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5\" (UID: \"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:22:53.662318 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.662292    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5\" (UID: \"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:22:53.663018 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.662998    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5\" (UID: \"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:22:53.680778 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.680744    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhcxv\" (UniqueName: \"kubernetes.io/projected/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-kube-api-access-vhcxv\") pod \"payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5\" (UID: \"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:22:53.849293 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:53.849258    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:22:54.156340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:54.156315    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"]
Oct 30 03:22:54.159254 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:22:54.159208    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03805bd7_ff01_4eb0_ae47_91d75b7c3ab6.slice/crio-0219c0d4d063055e6b3db19f0b78fe4ff8cfd0393a4915de43d272028f67ee0a WatchSource:0}: Error finding container 0219c0d4d063055e6b3db19f0b78fe4ff8cfd0393a4915de43d272028f67ee0a: Status 404 returned error can't find the container with id 0219c0d4d063055e6b3db19f0b78fe4ff8cfd0393a4915de43d272028f67ee0a
Oct 30 03:22:54.315576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:54.315538    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:54.589586 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:54.589553    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:54.599207 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:54.599177    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5" event={"ID":"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6","Type":"ContainerStarted","Data":"dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b"}
Oct 30 03:22:54.599369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:54.599259    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5" event={"ID":"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6","Type":"ContainerStarted","Data":"0219c0d4d063055e6b3db19f0b78fe4ff8cfd0393a4915de43d272028f67ee0a"}
Oct 30 03:22:54.599369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:54.599291    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:22:54.639437 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:54.639365    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5" podStartSLOduration=1.639352007 podStartE2EDuration="1.639352007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:54.636486332 +0000 UTC m=+2519.938664679" watchObservedRunningTime="2026-10-30 03:22:54.639352007 +0000 UTC m=+2519.941530370"
Oct 30 03:22:55.450118 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:22:55.450030    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:22:55.978253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:55.978220    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:56.120674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.120633    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:56.319873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.305544    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:56.349210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.349174    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"]
Oct 30 03:22:56.357981 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.357955    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:22:56.384091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.382892    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-948965\""
Oct 30 03:22:56.384091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.382977    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-948965-dockercfg-pxzmw\""
Oct 30 03:22:56.407683 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.407646    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"]
Oct 30 03:22:56.500253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.500218    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vjkw\" (UniqueName: \"kubernetes.io/projected/090b7c5e-0712-43bf-8205-7da384c1f38c-kube-api-access-9vjkw\") pod \"payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh\" (UID: \"090b7c5e-0712-43bf-8205-7da384c1f38c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:22:56.500500 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.500295    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b7c5e-0712-43bf-8205-7da384c1f38c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh\" (UID: \"090b7c5e-0712-43bf-8205-7da384c1f38c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:22:56.602371 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.602296    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b7c5e-0712-43bf-8205-7da384c1f38c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh\" (UID: \"090b7c5e-0712-43bf-8205-7da384c1f38c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:22:56.602519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.602507    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vjkw\" (UniqueName: \"kubernetes.io/projected/090b7c5e-0712-43bf-8205-7da384c1f38c-kube-api-access-9vjkw\") pod \"payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh\" (UID: \"090b7c5e-0712-43bf-8205-7da384c1f38c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:22:56.603050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.603029    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b7c5e-0712-43bf-8205-7da384c1f38c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh\" (UID: \"090b7c5e-0712-43bf-8205-7da384c1f38c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:22:56.621459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.621431    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vjkw\" (UniqueName: \"kubernetes.io/projected/090b7c5e-0712-43bf-8205-7da384c1f38c-kube-api-access-9vjkw\") pod \"payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh\" (UID: \"090b7c5e-0712-43bf-8205-7da384c1f38c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:22:56.686153 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:56.686123    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:22:57.013124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:57.013102    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"]
Oct 30 03:22:57.015590 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:22:57.015565    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090b7c5e_0712_43bf_8205_7da384c1f38c.slice/crio-de73d3c7cdc3e22e19b05d5b63a1444e99d8a92d345362724c24b31667f609fb WatchSource:0}: Error finding container de73d3c7cdc3e22e19b05d5b63a1444e99d8a92d345362724c24b31667f609fb: Status 404 returned error can't find the container with id de73d3c7cdc3e22e19b05d5b63a1444e99d8a92d345362724c24b31667f609fb
Oct 30 03:22:57.624461 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:57.624421    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh" event={"ID":"090b7c5e-0712-43bf-8205-7da384c1f38c","Type":"ContainerStarted","Data":"1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5"}
Oct 30 03:22:57.624634 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:57.624504    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh" event={"ID":"090b7c5e-0712-43bf-8205-7da384c1f38c","Type":"ContainerStarted","Data":"de73d3c7cdc3e22e19b05d5b63a1444e99d8a92d345362724c24b31667f609fb"}
Oct 30 03:22:57.624634 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:57.624533    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:22:57.662405 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:57.662329    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh" podStartSLOduration=1.662316154 podStartE2EDuration="1.662316154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:57.659899497 +0000 UTC m=+2522.962077862" watchObservedRunningTime="2026-10-30 03:22:57.662316154 +0000 UTC m=+2522.964494518"
Oct 30 03:22:58.225676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:22:58.225642    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:00.323181 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:00.323122    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:02.110191 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:02.110151    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:05.174961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:05.174920    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:05.611451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:05.611422    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:23:06.127680 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:06.127642    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:07.183817 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:07.183737    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:07.553270 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:07.553222    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:08.585218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:08.585176    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:08.634481 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:08.634457    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:23:08.756899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:08.753987    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:09.787007 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.786977    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"]
Oct 30 03:23:09.792822 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.792801    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:09.809455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.809432    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"]
Oct 30 03:23:09.815239 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.815213    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-734274\""
Oct 30 03:23:09.815341 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.815245    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-734274-dockercfg-lz7pj\""
Oct 30 03:23:09.858325 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.858286    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d250afa1-e7b2-4a48-965c-57fb24cb1929-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-734274-84746f954-z25tg\" (UID: \"d250afa1-e7b2-4a48-965c-57fb24cb1929\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:09.858510 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.858461    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjj7l\" (UniqueName: \"kubernetes.io/projected/d250afa1-e7b2-4a48-965c-57fb24cb1929-kube-api-access-vjj7l\") pod \"payload-processing-e2e-ipp-b-734274-84746f954-z25tg\" (UID: \"d250afa1-e7b2-4a48-965c-57fb24cb1929\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:09.933531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.933498    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"]
Oct 30 03:23:09.941520 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.941493    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:09.953327 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.953302    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"]
Oct 30 03:23:09.960353 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.960325    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjj7l\" (UniqueName: \"kubernetes.io/projected/d250afa1-e7b2-4a48-965c-57fb24cb1929-kube-api-access-vjj7l\") pod \"payload-processing-e2e-ipp-b-734274-84746f954-z25tg\" (UID: \"d250afa1-e7b2-4a48-965c-57fb24cb1929\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:09.960643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.960622    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d250afa1-e7b2-4a48-965c-57fb24cb1929-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-734274-84746f954-z25tg\" (UID: \"d250afa1-e7b2-4a48-965c-57fb24cb1929\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:09.961576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.961553    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d250afa1-e7b2-4a48-965c-57fb24cb1929-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-734274-84746f954-z25tg\" (UID: \"d250afa1-e7b2-4a48-965c-57fb24cb1929\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:09.980212 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:09.980189    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjj7l\" (UniqueName: \"kubernetes.io/projected/d250afa1-e7b2-4a48-965c-57fb24cb1929-kube-api-access-vjj7l\") pod \"payload-processing-e2e-ipp-b-734274-84746f954-z25tg\" (UID: \"d250afa1-e7b2-4a48-965c-57fb24cb1929\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:10.062206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.062175    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef60c73-abc4-403a-ba3a-62a7c87da7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd\" (UID: \"eef60c73-abc4-403a-ba3a-62a7c87da7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:10.062344 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.062236    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gbmq\" (UniqueName: \"kubernetes.io/projected/eef60c73-abc4-403a-ba3a-62a7c87da7b6-kube-api-access-7gbmq\") pod \"payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd\" (UID: \"eef60c73-abc4-403a-ba3a-62a7c87da7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:10.113140 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.113108    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:10.164267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.164231    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef60c73-abc4-403a-ba3a-62a7c87da7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd\" (UID: \"eef60c73-abc4-403a-ba3a-62a7c87da7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:10.164421 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.164325    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbmq\" (UniqueName: \"kubernetes.io/projected/eef60c73-abc4-403a-ba3a-62a7c87da7b6-kube-api-access-7gbmq\") pod \"payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd\" (UID: \"eef60c73-abc4-403a-ba3a-62a7c87da7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:10.165000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.164976    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef60c73-abc4-403a-ba3a-62a7c87da7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd\" (UID: \"eef60c73-abc4-403a-ba3a-62a7c87da7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:10.183089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.182937    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gbmq\" (UniqueName: \"kubernetes.io/projected/eef60c73-abc4-403a-ba3a-62a7c87da7b6-kube-api-access-7gbmq\") pod \"payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd\" (UID: \"eef60c73-abc4-403a-ba3a-62a7c87da7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:10.260542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.260511    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:10.377360 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.377324    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:10.449901 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.436658    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"]
Oct 30 03:23:10.564055 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.564027    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"]
Oct 30 03:23:10.566688 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:23:10.566662    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef60c73_abc4_403a_ba3a_62a7c87da7b6.slice/crio-cbde5bab2fd1c80bc903e3f63a7ab7fe8e354eb887d890163ee33f529d9b8e1e WatchSource:0}: Error finding container cbde5bab2fd1c80bc903e3f63a7ab7fe8e354eb887d890163ee33f529d9b8e1e: Status 404 returned error can't find the container with id cbde5bab2fd1c80bc903e3f63a7ab7fe8e354eb887d890163ee33f529d9b8e1e
Oct 30 03:23:10.809401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.809363    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg" event={"ID":"d250afa1-e7b2-4a48-965c-57fb24cb1929","Type":"ContainerStarted","Data":"6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6"}
Oct 30 03:23:10.809835 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.809431    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg" event={"ID":"d250afa1-e7b2-4a48-965c-57fb24cb1929","Type":"ContainerStarted","Data":"702c35be3b03656d0d779ef7767fe373f71e307851c16429ff29d036ae6e0592"}
Oct 30 03:23:10.809835 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.809457    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:10.811157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.811131    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd" event={"ID":"eef60c73-abc4-403a-ba3a-62a7c87da7b6","Type":"ContainerStarted","Data":"be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9"}
Oct 30 03:23:10.811288 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.811176    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd" event={"ID":"eef60c73-abc4-403a-ba3a-62a7c87da7b6","Type":"ContainerStarted","Data":"cbde5bab2fd1c80bc903e3f63a7ab7fe8e354eb887d890163ee33f529d9b8e1e"}
Oct 30 03:23:10.811288 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.811217    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:10.888669 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.888587    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg" podStartSLOduration=1.888573795 podStartE2EDuration="1.888573795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:10.884797324 +0000 UTC m=+2536.186975666" watchObservedRunningTime="2026-10-30 03:23:10.888573795 +0000 UTC m=+2536.190752159"
Oct 30 03:23:10.932506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:10.932463    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd" podStartSLOduration=1.9324498970000001 podStartE2EDuration="1.932449897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:10.931636573 +0000 UTC m=+2536.233814939" watchObservedRunningTime="2026-10-30 03:23:10.932449897 +0000 UTC m=+2536.234628262"
Oct 30 03:23:12.393284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:12.393250    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:13.973528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:13.973494    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:15.663613 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:15.663577    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:17.310178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:17.310134    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:17.432911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:17.432877    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:19.277073 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:19.277038    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:21.306131 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:21.306085    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:21.487450 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:21.487413    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:21.607584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:21.607505    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:21.825666 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:21.825634    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:21.831071 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:21.831039    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:22.099765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:22.099712    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:22.304812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:22.304733    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:22.517504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:22.517416    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:22.825101 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:22.825065    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:24.461569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:24.461526    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:25.447657 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:23:25.447622    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:23:26.204874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:26.204517    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:28.001721 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:28.001687    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:28.737455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:28.737415    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:28.860266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:28.860229    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:29.357230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:29.357162    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:29.691828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:29.691742    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"]
Oct 30 03:23:29.835351 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:29.835323    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:29.968209 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:29.968097    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:30.102663 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:30.102614    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:31.567779 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:31.567671    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:33.362905 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:33.362814    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:34.692732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:34.692694    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk" podUID="3d8676ca-1e01-46bd-a45f-45a13be5c888" containerName="payload-pre-processing" containerID="cri-o://d4bfa632ea42f39db38f9b9d6fb18e66020c534fc81c2abf9754730bbdb066cb" gracePeriod=25
Oct 30 03:23:34.868258 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:34.868226    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:35.006338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:35.006308    5061 generic.go:356] "Generic (PLEG): container finished" podID="3d8676ca-1e01-46bd-a45f-45a13be5c888" containerID="d4bfa632ea42f39db38f9b9d6fb18e66020c534fc81c2abf9754730bbdb066cb" exitCode=0
Oct 30 03:23:35.006484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:35.006378    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk" event={"ID":"3d8676ca-1e01-46bd-a45f-45a13be5c888","Type":"ContainerDied","Data":"d4bfa632ea42f39db38f9b9d6fb18e66020c534fc81c2abf9754730bbdb066cb"}
Oct 30 03:23:35.006484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:35.006429    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk" event={"ID":"3d8676ca-1e01-46bd-a45f-45a13be5c888","Type":"ContainerDied","Data":"cac673c7db7817f86549dd8820486c7ea8d5e0d04be624ea026f6c744e105bf1"}
Oct 30 03:23:35.006484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:35.006446    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cac673c7db7817f86549dd8820486c7ea8d5e0d04be624ea026f6c744e105bf1"
Oct 30 03:23:35.015947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:35.015921    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:23:35.052744 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:35.052720    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8676ca-1e01-46bd-a45f-45a13be5c888-kube-api-access-jt7d4\" (UniqueName: \"kubernetes.io/projected/3d8676ca-1e01-46bd-a45f-45a13be5c888-kube-api-access-jt7d4\") pod \"3d8676ca-1e01-46bd-a45f-45a13be5c888\" (UID: \"3d8676ca-1e01-46bd-a45f-45a13be5c888\") "
Oct 30 03:23:35.052938 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:35.052864    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8676ca-1e01-46bd-a45f-45a13be5c888-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8676ca-1e01-46bd-a45f-45a13be5c888-plugins-config-volume\") pod \"3d8676ca-1e01-46bd-a45f-45a13be5c888\" (UID: \"3d8676ca-1e01-46bd-a45f-45a13be5c888\") "
Oct 30 03:23:35.053245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:35.053222    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8676ca-1e01-46bd-a45f-45a13be5c888-plugins-config-volume" pod "3d8676ca-1e01-46bd-a45f-45a13be5c888" (UID: "3d8676ca-1e01-46bd-a45f-45a13be5c888"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:35.053491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:35.053470    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8676ca-1e01-46bd-a45f-45a13be5c888-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:23:35.057442 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:35.057418    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8676ca-1e01-46bd-a45f-45a13be5c888-kube-api-access-jt7d4" pod "3d8676ca-1e01-46bd-a45f-45a13be5c888" (UID: "3d8676ca-1e01-46bd-a45f-45a13be5c888"). InnerVolumeSpecName "kube-api-access-jt7d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:35.155344 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:35.155312    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt7d4\" (UniqueName: \"kubernetes.io/projected/3d8676ca-1e01-46bd-a45f-45a13be5c888-kube-api-access-jt7d4\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:23:36.013826 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:36.013793    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"
Oct 30 03:23:36.065242 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:36.065211    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"]
Oct 30 03:23:36.075510 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:36.075483    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a0f7e-654f749dbc-6lrjk"]
Oct 30 03:23:36.849742 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:36.849697    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8676ca-1e01-46bd-a45f-45a13be5c888" path="/var/lib/kubelet/pods/3d8676ca-1e01-46bd-a45f-45a13be5c888/volumes"
Oct 30 03:23:36.850835 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:36.850808    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:36.973342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:36.973314    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:37.121173 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:37.121090    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:38.945449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:38.945407    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:39.102459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:39.102424    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:40.882897 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:40.882802    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:42.626547 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:42.626512    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:44.343067 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:44.343005    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:46.021709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:46.021593    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:46.161015 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:46.160981    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:48.192526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:48.192491    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:49.397761 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:49.397691    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:49.972038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:49.972001    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:50.305801 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:50.305756    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:52.222024 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:52.221988    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:53.611400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.611363    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"]
Oct 30 03:23:53.613240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.613204    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv" podUID="27169abe-153c-4a03-94ed-4c247e23d698" containerName="maas-api" containerID="cri-o://429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a" gracePeriod=30
Oct 30 03:23:53.707065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.706988    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:53.715307 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.715277    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv" podUID="27169abe-153c-4a03-94ed-4c247e23d698" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.154:8443/health\": dial tcp 10.132.0.154:8443: connect: connection refused"
Oct 30 03:23:53.787313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.787279    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"]
Oct 30 03:23:53.787492 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.787465    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg" podUID="d250afa1-e7b2-4a48-965c-57fb24cb1929" containerName="payload-processing" containerID="cri-o://6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6" gracePeriod=30
Oct 30 03:23:53.823904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.822834    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"]
Oct 30 03:23:53.824303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.824202    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd" podUID="eef60c73-abc4-403a-ba3a-62a7c87da7b6" containerName="payload-pre-processing" containerID="cri-o://be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9" gracePeriod=30
Oct 30 03:23:53.926218 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.926182    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:53.972681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.972651    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:23:53.975095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.975060    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-maas-api-tls\") pod \"27169abe-153c-4a03-94ed-4c247e23d698\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") "
Oct 30 03:23:53.975234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.975113    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27169abe-153c-4a03-94ed-4c247e23d698-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27169abe-153c-4a03-94ed-4c247e23d698-openshift-service-ca\") pod \"27169abe-153c-4a03-94ed-4c247e23d698\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") "
Oct 30 03:23:53.975234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.975138    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27169abe-153c-4a03-94ed-4c247e23d698-kube-api-access-2slxv\" (UniqueName: \"kubernetes.io/projected/27169abe-153c-4a03-94ed-4c247e23d698-kube-api-access-2slxv\") pod \"27169abe-153c-4a03-94ed-4c247e23d698\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") "
Oct 30 03:23:53.975234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.975163    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-metrics-certs\") pod \"27169abe-153c-4a03-94ed-4c247e23d698\" (UID: \"27169abe-153c-4a03-94ed-4c247e23d698\") "
Oct 30 03:23:53.976352 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.976324    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27169abe-153c-4a03-94ed-4c247e23d698-openshift-service-ca" pod "27169abe-153c-4a03-94ed-4c247e23d698" (UID: "27169abe-153c-4a03-94ed-4c247e23d698"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:53.977930 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.977901    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-metrics-certs" pod "27169abe-153c-4a03-94ed-4c247e23d698" (UID: "27169abe-153c-4a03-94ed-4c247e23d698"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:23:53.978209 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.978170    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-maas-api-tls" pod "27169abe-153c-4a03-94ed-4c247e23d698" (UID: "27169abe-153c-4a03-94ed-4c247e23d698"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:23:53.979051 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:53.979018    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27169abe-153c-4a03-94ed-4c247e23d698-kube-api-access-2slxv" pod "27169abe-153c-4a03-94ed-4c247e23d698" (UID: "27169abe-153c-4a03-94ed-4c247e23d698"). InnerVolumeSpecName "kube-api-access-2slxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:54.077606 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.077573    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:23:54.077606 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.077603    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27169abe-153c-4a03-94ed-4c247e23d698-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:23:54.077870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.077618    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27169abe-153c-4a03-94ed-4c247e23d698-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:23:54.077870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.077628    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2slxv\" (UniqueName: \"kubernetes.io/projected/27169abe-153c-4a03-94ed-4c247e23d698-kube-api-access-2slxv\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:23:54.143757 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.143727    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:54.201871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.197082    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:54.241870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.237073    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d250afa1-e7b2-4a48-965c-57fb24cb1929-kube-api-access-vjj7l\" (UniqueName: \"kubernetes.io/projected/d250afa1-e7b2-4a48-965c-57fb24cb1929-kube-api-access-vjj7l\") pod \"d250afa1-e7b2-4a48-965c-57fb24cb1929\" (UID: \"d250afa1-e7b2-4a48-965c-57fb24cb1929\") "
Oct 30 03:23:54.241870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.237424    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d250afa1-e7b2-4a48-965c-57fb24cb1929-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d250afa1-e7b2-4a48-965c-57fb24cb1929-plugins-config-volume\") pod \"d250afa1-e7b2-4a48-965c-57fb24cb1929\" (UID: \"d250afa1-e7b2-4a48-965c-57fb24cb1929\") "
Oct 30 03:23:54.253643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.253611    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d250afa1-e7b2-4a48-965c-57fb24cb1929-plugins-config-volume" pod "d250afa1-e7b2-4a48-965c-57fb24cb1929" (UID: "d250afa1-e7b2-4a48-965c-57fb24cb1929"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:54.308879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.303609    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d250afa1-e7b2-4a48-965c-57fb24cb1929-kube-api-access-vjj7l" pod "d250afa1-e7b2-4a48-965c-57fb24cb1929" (UID: "d250afa1-e7b2-4a48-965c-57fb24cb1929"). InnerVolumeSpecName "kube-api-access-vjj7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:54.314398 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.312340    5061 generic.go:356] "Generic (PLEG): container finished" podID="d250afa1-e7b2-4a48-965c-57fb24cb1929" containerID="6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6" exitCode=0
Oct 30 03:23:54.314398 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.312540    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"
Oct 30 03:23:54.316008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.315978    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg" event={"ID":"d250afa1-e7b2-4a48-965c-57fb24cb1929","Type":"ContainerDied","Data":"6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6"}
Oct 30 03:23:54.316227 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.316206    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg" event={"ID":"d250afa1-e7b2-4a48-965c-57fb24cb1929","Type":"ContainerDied","Data":"702c35be3b03656d0d779ef7767fe373f71e307851c16429ff29d036ae6e0592"}
Oct 30 03:23:54.316359 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.316345    5061 scope.go:122] "RemoveContainer" containerID="6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6"
Oct 30 03:23:54.324444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.324416    5061 generic.go:356] "Generic (PLEG): container finished" podID="27169abe-153c-4a03-94ed-4c247e23d698" containerID="429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a" exitCode=0
Oct 30 03:23:54.324736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.324712    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv" event={"ID":"27169abe-153c-4a03-94ed-4c247e23d698","Type":"ContainerDied","Data":"429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a"}
Oct 30 03:23:54.325727 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.325701    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv" event={"ID":"27169abe-153c-4a03-94ed-4c247e23d698","Type":"ContainerDied","Data":"69278038bc9518f13d167edcf4a89faa4a68c8bd8753b8fed200f88a5caca07e"}
Oct 30 03:23:54.326918 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.326900    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"
Oct 30 03:23:54.336119 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.336094    5061 generic.go:356] "Generic (PLEG): container finished" podID="eef60c73-abc4-403a-ba3a-62a7c87da7b6" containerID="be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9" exitCode=0
Oct 30 03:23:54.336407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.336387    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd" event={"ID":"eef60c73-abc4-403a-ba3a-62a7c87da7b6","Type":"ContainerDied","Data":"be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9"}
Oct 30 03:23:54.336555 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.336533    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd" event={"ID":"eef60c73-abc4-403a-ba3a-62a7c87da7b6","Type":"ContainerDied","Data":"cbde5bab2fd1c80bc903e3f63a7ab7fe8e354eb887d890163ee33f529d9b8e1e"}
Oct 30 03:23:54.336906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.336886    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"
Oct 30 03:23:54.338699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.338673    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef60c73-abc4-403a-ba3a-62a7c87da7b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef60c73-abc4-403a-ba3a-62a7c87da7b6-plugins-config-volume\") pod \"eef60c73-abc4-403a-ba3a-62a7c87da7b6\" (UID: \"eef60c73-abc4-403a-ba3a-62a7c87da7b6\") "
Oct 30 03:23:54.338905 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.338885    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef60c73-abc4-403a-ba3a-62a7c87da7b6-kube-api-access-7gbmq\" (UniqueName: \"kubernetes.io/projected/eef60c73-abc4-403a-ba3a-62a7c87da7b6-kube-api-access-7gbmq\") pod \"eef60c73-abc4-403a-ba3a-62a7c87da7b6\" (UID: \"eef60c73-abc4-403a-ba3a-62a7c87da7b6\") "
Oct 30 03:23:54.339887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.339840    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d250afa1-e7b2-4a48-965c-57fb24cb1929-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:23:54.340018 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.340003    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjj7l\" (UniqueName: \"kubernetes.io/projected/d250afa1-e7b2-4a48-965c-57fb24cb1929-kube-api-access-vjj7l\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:23:54.342823 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.342800    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef60c73-abc4-403a-ba3a-62a7c87da7b6-plugins-config-volume" pod "eef60c73-abc4-403a-ba3a-62a7c87da7b6" (UID: "eef60c73-abc4-403a-ba3a-62a7c87da7b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:54.363293 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.363269    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef60c73-abc4-403a-ba3a-62a7c87da7b6-kube-api-access-7gbmq" pod "eef60c73-abc4-403a-ba3a-62a7c87da7b6" (UID: "eef60c73-abc4-403a-ba3a-62a7c87da7b6"). InnerVolumeSpecName "kube-api-access-7gbmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:54.369380 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.369355    5061 scope.go:122] "RemoveContainer" containerID="6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6"
Oct 30 03:23:54.370528 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:23:54.370499    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6\": container with ID starting with 6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6 not found: ID does not exist" containerID="6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6"
Oct 30 03:23:54.370641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.370537    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6"} err="failed to get container status \"6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6\": rpc error: code = NotFound desc = could not find container \"6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6\": container with ID starting with 6434d13b08c5b9ca362a7671d7e411052f6db2d4f6027734943d987e88f000b6 not found: ID does not exist"
Oct 30 03:23:54.370641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.370560    5061 scope.go:122] "RemoveContainer" containerID="429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a"
Oct 30 03:23:54.393187 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.393158    5061 scope.go:122] "RemoveContainer" containerID="429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a"
Oct 30 03:23:54.394135 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:23:54.394103    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a\": container with ID starting with 429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a not found: ID does not exist" containerID="429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a"
Oct 30 03:23:54.394316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.394284    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a"} err="failed to get container status \"429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a\": rpc error: code = NotFound desc = could not find container \"429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a\": container with ID starting with 429d994e2f0320d9d6f60a7df5ee4388f958035c39ae5b229c08755b5ef7231a not found: ID does not exist"
Oct 30 03:23:54.394434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.394420    5061 scope.go:122] "RemoveContainer" containerID="be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9"
Oct 30 03:23:54.409408 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.409375    5061 scope.go:122] "RemoveContainer" containerID="be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9"
Oct 30 03:23:54.409748 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:23:54.409724    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9\": container with ID starting with be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9 not found: ID does not exist" containerID="be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9"
Oct 30 03:23:54.409829 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.409758    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9"} err="failed to get container status \"be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9\": rpc error: code = NotFound desc = could not find container \"be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9\": container with ID starting with be7dbaf4f684f9ae58a45b324e6477a78868a329ac1af0668005531e45816be9 not found: ID does not exist"
Oct 30 03:23:54.413595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.413559    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"]
Oct 30 03:23:54.421505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.421483    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-734274-84746f954-z25tg"]
Oct 30 03:23:54.441599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.441571    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef60c73-abc4-403a-ba3a-62a7c87da7b6-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:23:54.441599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.441597    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gbmq\" (UniqueName: \"kubernetes.io/projected/eef60c73-abc4-403a-ba3a-62a7c87da7b6-kube-api-access-7gbmq\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:23:54.450643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.450615    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:54.478822 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.478795    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"]
Oct 30 03:23:54.492525 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.492496    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-734274-549ffb8cc7-44ntv"]
Oct 30 03:23:54.596511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.596484    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:54.700338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.700309    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"]
Oct 30 03:23:54.704156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.704132    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-734274-758c649cdc-87sfd"]
Oct 30 03:23:54.849820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.849792    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27169abe-153c-4a03-94ed-4c247e23d698" path="/var/lib/kubelet/pods/27169abe-153c-4a03-94ed-4c247e23d698/volumes"
Oct 30 03:23:54.850231 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.850216    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d250afa1-e7b2-4a48-965c-57fb24cb1929" path="/var/lib/kubelet/pods/d250afa1-e7b2-4a48-965c-57fb24cb1929/volumes"
Oct 30 03:23:54.850601 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.850589    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef60c73-abc4-403a-ba3a-62a7c87da7b6" path="/var/lib/kubelet/pods/eef60c73-abc4-403a-ba3a-62a7c87da7b6/volumes"
Oct 30 03:23:54.879632 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:54.879587    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:55.098933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:55.098879    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:55.448837 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:23:55.448741    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:23:56.818942 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:56.818908    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:58.684520 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:23:58.684475    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:00.123448 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:00.123413    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:00.685592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:00.685546    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:00.859529 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:00.859485    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:01.018164 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:01.018043    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:01.439080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:01.439048    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:01.981330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:01.981297    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:02.137178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:02.137137    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:02.347013 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:02.346981    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:02.793667 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:02.793638    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:03.071147 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:03.071065    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:05.879037 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:05.878995    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:07.191475 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:07.191440    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:07.402872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:07.388056    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:07.751021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:07.750978    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"]
Oct 30 03:24:07.908764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:07.908678    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:08.084451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.084366    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:08.231536 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.231505    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:08.393984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.393894    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"]
Oct 30 03:24:08.395110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395080    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d250afa1-e7b2-4a48-965c-57fb24cb1929" containerName="payload-processing"
Oct 30 03:24:08.395110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395109    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d250afa1-e7b2-4a48-965c-57fb24cb1929" containerName="payload-processing"
Oct 30 03:24:08.395329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395156    5061 cpu_manager.go:414] "containerMap: removing container" podUID="27169abe-153c-4a03-94ed-4c247e23d698" containerName="maas-api"
Oct 30 03:24:08.395329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395166    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27169abe-153c-4a03-94ed-4c247e23d698" containerName="maas-api"
Oct 30 03:24:08.395329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395182    5061 cpu_manager.go:414] "containerMap: removing container" podUID="3d8676ca-1e01-46bd-a45f-45a13be5c888" containerName="payload-pre-processing"
Oct 30 03:24:08.395329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395192    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8676ca-1e01-46bd-a45f-45a13be5c888" containerName="payload-pre-processing"
Oct 30 03:24:08.395329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395255    5061 cpu_manager.go:414] "containerMap: removing container" podUID="eef60c73-abc4-403a-ba3a-62a7c87da7b6" containerName="payload-pre-processing"
Oct 30 03:24:08.395329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395264    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef60c73-abc4-403a-ba3a-62a7c87da7b6" containerName="payload-pre-processing"
Oct 30 03:24:08.395677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395449    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef60c73-abc4-403a-ba3a-62a7c87da7b6" containerName="payload-pre-processing"
Oct 30 03:24:08.395677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395469    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d250afa1-e7b2-4a48-965c-57fb24cb1929" containerName="payload-processing"
Oct 30 03:24:08.395677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395486    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="27169abe-153c-4a03-94ed-4c247e23d698" containerName="maas-api"
Oct 30 03:24:08.395677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.395495    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8676ca-1e01-46bd-a45f-45a13be5c888" containerName="payload-pre-processing"
Oct 30 03:24:08.402605 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.402580    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"]
Oct 30 03:24:08.402773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.402709    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:24:08.406301 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.404093    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerName="istio-proxy" containerID="cri-o://9a2c597baa8d38e0a1a2f9dab569ac920dacd2ad99f886384578238f9a4d1878" gracePeriod=30
Oct 30 03:24:08.408515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.408203    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"]
Oct 30 03:24:08.408515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.408389    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" podUID="af43e5ce-2da5-4f93-9296-66e75df36e82" containerName="istio-proxy" containerID="cri-o://a3ee567cbe37cb91a365b4a79b95f8b79ac44f054b01740e6ccb024d676b193b" gracePeriod=30
Oct 30 03:24:08.428869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.428339    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a0f7e-dockercfg-ncnr4\""
Oct 30 03:24:08.432885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.432841    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a0f7e\""
Oct 30 03:24:08.432980 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.432881    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"]
Oct 30 03:24:08.433121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.433089    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerName="istio-proxy" containerID="cri-o://ac9ce0bb35418fc711a4ecc8d049b98780ef38b504c8232f9adddb7416a928d3" gracePeriod=30
Oct 30 03:24:08.438919 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.437229    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"]
Oct 30 03:24:08.453666 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.453618    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl6t9\" (UniqueName: \"kubernetes.io/projected/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-kube-api-access-sl6t9\") pod \"payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5\" (UID: \"b997ff8f-09d5-44e8-8c66-3c89ff54b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:24:08.453804 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.453783    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5\" (UID: \"b997ff8f-09d5-44e8-8c66-3c89ff54b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:24:08.479750 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.479724    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:08.628964 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.628891    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl6t9\" (UniqueName: \"kubernetes.io/projected/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-kube-api-access-sl6t9\") pod \"payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5\" (UID: \"b997ff8f-09d5-44e8-8c66-3c89ff54b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:24:08.629178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.629064    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5\" (UID: \"b997ff8f-09d5-44e8-8c66-3c89ff54b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:24:08.630491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.630373    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5\" (UID: \"b997ff8f-09d5-44e8-8c66-3c89ff54b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:24:08.649028 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.648964    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl6t9\" (UniqueName: \"kubernetes.io/projected/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-kube-api-access-sl6t9\") pod \"payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5\" (UID: \"b997ff8f-09d5-44e8-8c66-3c89ff54b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:24:08.728716 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:08.728683    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:24:09.018789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:09.018703    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:09.044678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:09.044638    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"]
Oct 30 03:24:09.046263 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:24:09.046233    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb997ff8f_09d5_44e8_8c66_3c89ff54b04f.slice/crio-18834286eeee1587ceff567843451ac9d3303fde4c3db3ad96a778a3891f6732 WatchSource:0}: Error finding container 18834286eeee1587ceff567843451ac9d3303fde4c3db3ad96a778a3891f6732: Status 404 returned error can't find the container with id 18834286eeee1587ceff567843451ac9d3303fde4c3db3ad96a778a3891f6732
Oct 30 03:24:09.648177 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:09.648140    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5" event={"ID":"b997ff8f-09d5-44e8-8c66-3c89ff54b04f","Type":"ContainerStarted","Data":"0de491ddf8444d1048051b74fbf7815980b09bc787a18e9a7d6069f2fb4347cc"}
Oct 30 03:24:09.648678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:09.648194    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5" event={"ID":"b997ff8f-09d5-44e8-8c66-3c89ff54b04f","Type":"ContainerStarted","Data":"18834286eeee1587ceff567843451ac9d3303fde4c3db3ad96a778a3891f6732"}
Oct 30 03:24:09.648678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:09.648262    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:24:09.689782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:09.689731    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5" podStartSLOduration=1.689718601 podStartE2EDuration="1.689718601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:09.687059421 +0000 UTC m=+2594.989237800" watchObservedRunningTime="2026-10-30 03:24:09.689718601 +0000 UTC m=+2594.991896965"
Oct 30 03:24:11.012834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:11.012794    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:12.871911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:12.871876    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:13.401662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.401627    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"]
Oct 30 03:24:13.551975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.551933    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:13.679333 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.678184    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:13.692405 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.692372    5061 generic.go:356] "Generic (PLEG): container finished" podID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerID="ac9ce0bb35418fc711a4ecc8d049b98780ef38b504c8232f9adddb7416a928d3" exitCode=0
Oct 30 03:24:13.693975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.693293    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" event={"ID":"93c3171d-e418-4df4-b4dc-aa9a8ced8706","Type":"ContainerDied","Data":"ac9ce0bb35418fc711a4ecc8d049b98780ef38b504c8232f9adddb7416a928d3"}
Oct 30 03:24:13.699713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.697566    5061 generic.go:356] "Generic (PLEG): container finished" podID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerID="9a2c597baa8d38e0a1a2f9dab569ac920dacd2ad99f886384578238f9a4d1878" exitCode=0
Oct 30 03:24:13.699713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.697654    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" event={"ID":"a9bd36df-8df5-4293-9a2a-efa4da29d75d","Type":"ContainerDied","Data":"9a2c597baa8d38e0a1a2f9dab569ac920dacd2ad99f886384578238f9a4d1878"}
Oct 30 03:24:13.706761 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.706567    5061 generic.go:356] "Generic (PLEG): container finished" podID="af43e5ce-2da5-4f93-9296-66e75df36e82" containerID="a3ee567cbe37cb91a365b4a79b95f8b79ac44f054b01740e6ccb024d676b193b" exitCode=0
Oct 30 03:24:13.706936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.706773    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" event={"ID":"af43e5ce-2da5-4f93-9296-66e75df36e82","Type":"ContainerDied","Data":"a3ee567cbe37cb91a365b4a79b95f8b79ac44f054b01740e6ccb024d676b193b"}
Oct 30 03:24:13.782797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.781493    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:24:13.805969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.805938    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:24:13.809699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.809665    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-envoy\") pod \"af43e5ce-2da5-4f93-9296-66e75df36e82\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") "
Oct 30 03:24:13.809889 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.809735    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-kube-api-access-rb6fd\" (UniqueName: \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-kube-api-access-rb6fd\") pod \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") "
Oct 30 03:24:13.809889 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.809803    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-socket\") pod \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") "
Oct 30 03:24:13.810037 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.809892    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istiod-ca-cert\") pod \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") "
Oct 30 03:24:13.810037 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.809970    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-kube-api-access-xdrg6\" (UniqueName: \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-kube-api-access-xdrg6\") pod \"af43e5ce-2da5-4f93-9296-66e75df36e82\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") "
Oct 30 03:24:13.810162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.810039    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-token\" (UniqueName: \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-token\") pod \"af43e5ce-2da5-4f93-9296-66e75df36e82\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") "
Oct 30 03:24:13.810162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.810096    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-credential-socket\") pod \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") "
Oct 30 03:24:13.810162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.810124    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-certs\") pod \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") "
Oct 30 03:24:13.811472 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.811439    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istiod-ca-cert" pod "a9bd36df-8df5-4293-9a2a-efa4da29d75d" (UID: "a9bd36df-8df5-4293-9a2a-efa4da29d75d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:13.813459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.813432    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af43e5ce-2da5-4f93-9296-66e75df36e82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af43e5ce-2da5-4f93-9296-66e75df36e82-istiod-ca-cert\") pod \"af43e5ce-2da5-4f93-9296-66e75df36e82\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") "
Oct 30 03:24:13.813616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.813486    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-credential-socket\") pod \"af43e5ce-2da5-4f93-9296-66e75df36e82\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") "
Oct 30 03:24:13.813616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.813514    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-token\" (UniqueName: \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-token\") pod \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") "
Oct 30 03:24:13.813744 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.813633    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-envoy\") pod \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") "
Oct 30 03:24:13.813744 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.813689    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-certs\") pod \"af43e5ce-2da5-4f93-9296-66e75df36e82\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") "
Oct 30 03:24:13.813880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.813767    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-data\") pod \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") "
Oct 30 03:24:13.813880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.813842    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-socket\") pod \"af43e5ce-2da5-4f93-9296-66e75df36e82\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") "
Oct 30 03:24:13.814012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.813887    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-data\") pod \"af43e5ce-2da5-4f93-9296-66e75df36e82\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") "
Oct 30 03:24:13.814012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.813939    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-podinfo\") pod \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\" (UID: \"a9bd36df-8df5-4293-9a2a-efa4da29d75d\") "
Oct 30 03:24:13.814012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.813970    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-podinfo\") pod \"af43e5ce-2da5-4f93-9296-66e75df36e82\" (UID: \"af43e5ce-2da5-4f93-9296-66e75df36e82\") "
Oct 30 03:24:13.814653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.814625    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-certs" pod "a9bd36df-8df5-4293-9a2a-efa4da29d75d" (UID: "a9bd36df-8df5-4293-9a2a-efa4da29d75d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.814810 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.814783    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-socket" pod "a9bd36df-8df5-4293-9a2a-efa4da29d75d" (UID: "a9bd36df-8df5-4293-9a2a-efa4da29d75d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.814918 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.814886    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-envoy" pod "af43e5ce-2da5-4f93-9296-66e75df36e82" (UID: "af43e5ce-2da5-4f93-9296-66e75df36e82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.815120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.815100    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-credential-socket" pod "af43e5ce-2da5-4f93-9296-66e75df36e82" (UID: "af43e5ce-2da5-4f93-9296-66e75df36e82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.815201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.815178    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-data" pod "af43e5ce-2da5-4f93-9296-66e75df36e82" (UID: "af43e5ce-2da5-4f93-9296-66e75df36e82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.815274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.815216    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af43e5ce-2da5-4f93-9296-66e75df36e82-istiod-ca-cert" pod "af43e5ce-2da5-4f93-9296-66e75df36e82" (UID: "af43e5ce-2da5-4f93-9296-66e75df36e82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:13.815805 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.815778    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-data" pod "a9bd36df-8df5-4293-9a2a-efa4da29d75d" (UID: "a9bd36df-8df5-4293-9a2a-efa4da29d75d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.816128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.816074    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af43e5ce-2da5-4f93-9296-66e75df36e82-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.816221 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.816157    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-credential-socket" pod "a9bd36df-8df5-4293-9a2a-efa4da29d75d" (UID: "a9bd36df-8df5-4293-9a2a-efa4da29d75d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.816414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.816296    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.816414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.816325    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.816414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.816340    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.816414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.816354    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.816414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.816367    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.816414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.816380    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.816414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.816394    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.819420 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.819393    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-socket" pod "af43e5ce-2da5-4f93-9296-66e75df36e82" (UID: "af43e5ce-2da5-4f93-9296-66e75df36e82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.819528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.819458    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-certs" pod "af43e5ce-2da5-4f93-9296-66e75df36e82" (UID: "af43e5ce-2da5-4f93-9296-66e75df36e82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.820506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.820478    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-token" pod "af43e5ce-2da5-4f93-9296-66e75df36e82" (UID: "af43e5ce-2da5-4f93-9296-66e75df36e82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.821017 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.820975    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-podinfo" pod "a9bd36df-8df5-4293-9a2a-efa4da29d75d" (UID: "a9bd36df-8df5-4293-9a2a-efa4da29d75d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:13.830429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.830401    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-envoy" pod "a9bd36df-8df5-4293-9a2a-efa4da29d75d" (UID: "a9bd36df-8df5-4293-9a2a-efa4da29d75d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.830429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.830417    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-podinfo" pod "af43e5ce-2da5-4f93-9296-66e75df36e82" (UID: "af43e5ce-2da5-4f93-9296-66e75df36e82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:13.832239 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.832214    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-kube-api-access-rb6fd" pod "a9bd36df-8df5-4293-9a2a-efa4da29d75d" (UID: "a9bd36df-8df5-4293-9a2a-efa4da29d75d"). InnerVolumeSpecName "kube-api-access-rb6fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.832361 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.832248    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-kube-api-access-xdrg6" pod "af43e5ce-2da5-4f93-9296-66e75df36e82" (UID: "af43e5ce-2da5-4f93-9296-66e75df36e82"). InnerVolumeSpecName "kube-api-access-xdrg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.834191 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.834168    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-token" pod "a9bd36df-8df5-4293-9a2a-efa4da29d75d" (UID: "a9bd36df-8df5-4293-9a2a-efa4da29d75d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.875007 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.874977    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:24:13.917193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.917156    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-token\" (UniqueName: \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-token\") pod \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") "
Oct 30 03:24:13.917352 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.917224    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-envoy\") pod \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") "
Oct 30 03:24:13.917352 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.917278    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istiod-ca-cert\") pod \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") "
Oct 30 03:24:13.917522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.917354    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-kube-api-access-psrqr\" (UniqueName: \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-kube-api-access-psrqr\") pod \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") "
Oct 30 03:24:13.917522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.917415    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-data\") pod \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") "
Oct 30 03:24:13.917522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.917495    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-socket\") pod \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") "
Oct 30 03:24:13.917688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.917654    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-credential-socket\") pod \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") "
Oct 30 03:24:13.917747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.917684    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-podinfo\") pod \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") "
Oct 30 03:24:13.917811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.917743    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-certs\") pod \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\" (UID: \"93c3171d-e418-4df4-b4dc-aa9a8ced8706\") "
Oct 30 03:24:13.918373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.918194    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-socket" pod "93c3171d-e418-4df4-b4dc-aa9a8ced8706" (UID: "93c3171d-e418-4df4-b4dc-aa9a8ced8706"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.919433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.918559    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-data" pod "93c3171d-e418-4df4-b4dc-aa9a8ced8706" (UID: "93c3171d-e418-4df4-b4dc-aa9a8ced8706"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.919433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.918725    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-certs" pod "93c3171d-e418-4df4-b4dc-aa9a8ced8706" (UID: "93c3171d-e418-4df4-b4dc-aa9a8ced8706"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.919433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.918745    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-credential-socket" pod "93c3171d-e418-4df4-b4dc-aa9a8ced8706" (UID: "93c3171d-e418-4df4-b4dc-aa9a8ced8706"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.920389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920367    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920394    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920409    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920421    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920434    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920446    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920460    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43e5ce-2da5-4f93-9296-66e75df36e82-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920473    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bd36df-8df5-4293-9a2a-efa4da29d75d-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920486    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920500    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb6fd\" (UniqueName: \"kubernetes.io/projected/a9bd36df-8df5-4293-9a2a-efa4da29d75d-kube-api-access-rb6fd\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920513    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920525    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920537    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdrg6\" (UniqueName: \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-kube-api-access-xdrg6\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.920549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.920549    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af43e5ce-2da5-4f93-9296-66e75df36e82-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:13.921242 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.921131    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istiod-ca-cert" pod "93c3171d-e418-4df4-b4dc-aa9a8ced8706" (UID: "93c3171d-e418-4df4-b4dc-aa9a8ced8706"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:13.930553 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.930443    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-kube-api-access-psrqr" pod "93c3171d-e418-4df4-b4dc-aa9a8ced8706" (UID: "93c3171d-e418-4df4-b4dc-aa9a8ced8706"). InnerVolumeSpecName "kube-api-access-psrqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.931138 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.931111    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-envoy" pod "93c3171d-e418-4df4-b4dc-aa9a8ced8706" (UID: "93c3171d-e418-4df4-b4dc-aa9a8ced8706"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:13.931235 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.931178    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-token" pod "93c3171d-e418-4df4-b4dc-aa9a8ced8706" (UID: "93c3171d-e418-4df4-b4dc-aa9a8ced8706"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:13.931457 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:13.931437    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-podinfo" pod "93c3171d-e418-4df4-b4dc-aa9a8ced8706" (UID: "93c3171d-e418-4df4-b4dc-aa9a8ced8706"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:14.021906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.021837    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.021906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.021900    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.021906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.021913    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.022146 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.021926    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c3171d-e418-4df4-b4dc-aa9a8ced8706-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.022146 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.021939    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psrqr\" (UniqueName: \"kubernetes.io/projected/93c3171d-e418-4df4-b4dc-aa9a8ced8706-kube-api-access-psrqr\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:14.502709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.502675    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:14.716305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.716211    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"
Oct 30 03:24:14.716305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.716212    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h" event={"ID":"a9bd36df-8df5-4293-9a2a-efa4da29d75d","Type":"ContainerDied","Data":"99619ed436c8865bc2c6275a35097e46bcae55e60a4e330598a79ddb72f982b6"}
Oct 30 03:24:14.716515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.716350    5061 scope.go:122] "RemoveContainer" containerID="9a2c597baa8d38e0a1a2f9dab569ac920dacd2ad99f886384578238f9a4d1878"
Oct 30 03:24:14.717794 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.717761    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"
Oct 30 03:24:14.717896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.717762    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k" event={"ID":"af43e5ce-2da5-4f93-9296-66e75df36e82","Type":"ContainerDied","Data":"58df43aa40d1f8f65dd4618326cb5f512cb6d54582314e7b949617e1467b6053"}
Oct 30 03:24:14.719704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.719681    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"
Oct 30 03:24:14.719827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.719706    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b" event={"ID":"93c3171d-e418-4df4-b4dc-aa9a8ced8706","Type":"ContainerDied","Data":"7e892e282f0c4603193f1edeb7c9803598635752525f9998b08c2c828825740b"}
Oct 30 03:24:14.740622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.740600    5061 scope.go:122] "RemoveContainer" containerID="a3ee567cbe37cb91a365b4a79b95f8b79ac44f054b01740e6ccb024d676b193b"
Oct 30 03:24:14.751039 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.751010    5061 scope.go:122] "RemoveContainer" containerID="ac9ce0bb35418fc711a4ecc8d049b98780ef38b504c8232f9adddb7416a928d3"
Oct 30 03:24:14.776238 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.776211    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"]
Oct 30 03:24:14.786008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.785982    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-b7c9h"]
Oct 30 03:24:14.812763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.812733    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"]
Oct 30 03:24:14.817888 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.817868    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-wjs4k"]
Oct 30 03:24:14.853753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.853728    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" path="/var/lib/kubelet/pods/a9bd36df-8df5-4293-9a2a-efa4da29d75d/volumes"
Oct 30 03:24:14.854229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.854215    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af43e5ce-2da5-4f93-9296-66e75df36e82" path="/var/lib/kubelet/pods/af43e5ce-2da5-4f93-9296-66e75df36e82/volumes"
Oct 30 03:24:14.854515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.854501    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"]
Oct 30 03:24:14.854565 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:14.854527    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9245d1-openshift-default-5b5f444c7-rbz7b"]
Oct 30 03:24:16.087831 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:16.087794    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:16.849765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:16.849728    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" path="/var/lib/kubelet/pods/93c3171d-e418-4df4-b4dc-aa9a8ced8706/volumes"
Oct 30 03:24:17.772157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:17.772126    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:18.401936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:18.401899    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs" podUID="e44c0f23-392b-4bee-8c03-82f0b23abe90" containerName="payload-processing" containerID="cri-o://56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6" gracePeriod=25
Oct 30 03:24:18.989972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:18.989939    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:19.439302 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:19.439238    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:19.678677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:19.678642    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"]
Oct 30 03:24:19.729887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:19.729800    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"]
Oct 30 03:24:19.848286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:19.848257    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:20.060592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:20.060560    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:20.658592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:20.658561    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:24:21.622157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:21.622124    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:22.776073 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:22.776036    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main" containerID="cri-o://e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d" gracePeriod=45
Oct 30 03:24:23.181732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.181708    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 03:24:23.264594 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.264556    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-home\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-home\") pod \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") "
Oct 30 03:24:23.264977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.264664    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tmp-dir\") pod \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") "
Oct 30 03:24:23.264977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.264716    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-model-cache\") pod \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") "
Oct 30 03:24:23.264977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.264817    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e076698c-e0eb-42d9-bdd9-2ecd3318da04-kube-api-access-88bml\" (UniqueName: \"kubernetes.io/projected/e076698c-e0eb-42d9-bdd9-2ecd3318da04-kube-api-access-88bml\") pod \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") "
Oct 30 03:24:23.264977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.264887    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-dshm\") pod \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") "
Oct 30 03:24:23.265244 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.265011    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tls-certs\" (UniqueName: \"kubernetes.io/secret/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tls-certs\") pod \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\" (UID: \"e076698c-e0eb-42d9-bdd9-2ecd3318da04\") "
Oct 30 03:24:23.265244 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.265160    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tmp-dir" pod "e076698c-e0eb-42d9-bdd9-2ecd3318da04" (UID: "e076698c-e0eb-42d9-bdd9-2ecd3318da04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.265244 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.265196    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-home" pod "e076698c-e0eb-42d9-bdd9-2ecd3318da04" (UID: "e076698c-e0eb-42d9-bdd9-2ecd3318da04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.265412 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.265373    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-model-cache" pod "e076698c-e0eb-42d9-bdd9-2ecd3318da04" (UID: "e076698c-e0eb-42d9-bdd9-2ecd3318da04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.265717 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.265688    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.265781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.265725    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.265781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.265742    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.267972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.267942    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-dshm" pod "e076698c-e0eb-42d9-bdd9-2ecd3318da04" (UID: "e076698c-e0eb-42d9-bdd9-2ecd3318da04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:23.268286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.268261    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e076698c-e0eb-42d9-bdd9-2ecd3318da04-kube-api-access-88bml" pod "e076698c-e0eb-42d9-bdd9-2ecd3318da04" (UID: "e076698c-e0eb-42d9-bdd9-2ecd3318da04"). InnerVolumeSpecName "kube-api-access-88bml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:23.268610 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.268591    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tls-certs" pod "e076698c-e0eb-42d9-bdd9-2ecd3318da04" (UID: "e076698c-e0eb-42d9-bdd9-2ecd3318da04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:23.333403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.333375    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:23.368265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.368229    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88bml\" (UniqueName: \"kubernetes.io/projected/e076698c-e0eb-42d9-bdd9-2ecd3318da04-kube-api-access-88bml\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.368265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.368266    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e076698c-e0eb-42d9-bdd9-2ecd3318da04-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.368595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.368282    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e076698c-e0eb-42d9-bdd9-2ecd3318da04-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:23.790984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.790948    5061 generic.go:356] "Generic (PLEG): container finished" podID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerID="e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d" exitCode=0
Oct 30 03:24:23.791445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.790995    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"
Oct 30 03:24:23.791445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.791028    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" event={"ID":"e076698c-e0eb-42d9-bdd9-2ecd3318da04","Type":"ContainerDied","Data":"e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d"}
Oct 30 03:24:23.791445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.791074    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4" event={"ID":"e076698c-e0eb-42d9-bdd9-2ecd3318da04","Type":"ContainerDied","Data":"effeade8bc2821d459ea5e08b79e7f49c3a38367bced3fd2fc167543b60fb3a3"}
Oct 30 03:24:23.791445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.791093    5061 scope.go:122] "RemoveContainer" containerID="e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d"
Oct 30 03:24:23.802763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.802741    5061 scope.go:122] "RemoveContainer" containerID="e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d"
Oct 30 03:24:23.803048 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:24:23.803028    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d\": container with ID starting with e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d not found: ID does not exist" containerID="e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d"
Oct 30 03:24:23.803117 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.803058    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d"} err="failed to get container status \"e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d\": rpc error: code = NotFound desc = could not find container \"e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d\": container with ID starting with e0ddae6ca17843a2626a5b303fdc7b07b28b41cea25a862dbefd93a666538a0d not found: ID does not exist"
Oct 30 03:24:23.845395 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.845352    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"]
Oct 30 03:24:23.859142 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:23.859115    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-810119/simulator-w2-810119-kserve-6c8cdd7459-m22t4"]
Oct 30 03:24:24.679057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:24.679017    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2" podUID="0dbca485-e20b-4e49-9aa4-3cac4eb994c9" containerName="payload-processing" containerID="cri-o://74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41" gracePeriod=25
Oct 30 03:24:24.731012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:24.730973    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg" podUID="bc979a77-665d-40c3-85f7-66313483e56a" containerName="payload-pre-processing" containerID="cri-o://0b2dfb54d9662accc60a23910772b4f62aa56fce1a82a09996ad4414777216dc" gracePeriod=25
Oct 30 03:24:24.849318 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:24.849286    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" path="/var/lib/kubelet/pods/e076698c-e0eb-42d9-bdd9-2ecd3318da04/volumes"
Oct 30 03:24:25.433867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:25.433361    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:25.466557 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:24:25.466514    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:24:27.079455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:27.079420    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:28.179485 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:28.179449    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs" podUID="e44c0f23-392b-4bee-8c03-82f0b23abe90" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:24:28.718391 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:28.718352    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:28.952185 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:28.952143    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:29.190932 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:29.190896    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:29.712544 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:29.712511    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:29.892754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:29.892685    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"]
Oct 30 03:24:30.065447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:30.065413    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:30.330523 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:30.330445    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:30.685053 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:30.684974    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:31.024458 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:31.024361    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:33.519319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:33.519281    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:34.422129 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:34.422097    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:34.584553 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:34.584509    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2" podUID="0dbca485-e20b-4e49-9aa4-3cac4eb994c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:24:34.584982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:34.584518    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg" podUID="bc979a77-665d-40c3-85f7-66313483e56a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:24:34.714759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:34.714683    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:34.894317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:34.894279    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh" podUID="090b7c5e-0712-43bf-8205-7da384c1f38c" containerName="payload-processing" containerID="cri-o://1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5" gracePeriod=25
Oct 30 03:24:35.194808 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.194777    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:24:35.311319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.311290    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090b7c5e-0712-43bf-8205-7da384c1f38c-kube-api-access-9vjkw\" (UniqueName: \"kubernetes.io/projected/090b7c5e-0712-43bf-8205-7da384c1f38c-kube-api-access-9vjkw\") pod \"090b7c5e-0712-43bf-8205-7da384c1f38c\" (UID: \"090b7c5e-0712-43bf-8205-7da384c1f38c\") "
Oct 30 03:24:35.311523 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.311447    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090b7c5e-0712-43bf-8205-7da384c1f38c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b7c5e-0712-43bf-8205-7da384c1f38c-plugins-config-volume\") pod \"090b7c5e-0712-43bf-8205-7da384c1f38c\" (UID: \"090b7c5e-0712-43bf-8205-7da384c1f38c\") "
Oct 30 03:24:35.311924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.311890    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090b7c5e-0712-43bf-8205-7da384c1f38c-plugins-config-volume" pod "090b7c5e-0712-43bf-8205-7da384c1f38c" (UID: "090b7c5e-0712-43bf-8205-7da384c1f38c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:35.312190 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.312170    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b7c5e-0712-43bf-8205-7da384c1f38c-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:35.313557 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.313535    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090b7c5e-0712-43bf-8205-7da384c1f38c-kube-api-access-9vjkw" pod "090b7c5e-0712-43bf-8205-7da384c1f38c" (UID: "090b7c5e-0712-43bf-8205-7da384c1f38c"). InnerVolumeSpecName "kube-api-access-9vjkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:35.390891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.390829    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:35.413730 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.413702    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vjkw\" (UniqueName: \"kubernetes.io/projected/090b7c5e-0712-43bf-8205-7da384c1f38c-kube-api-access-9vjkw\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:35.602760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.602681    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:35.917303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.917218    5061 generic.go:356] "Generic (PLEG): container finished" podID="090b7c5e-0712-43bf-8205-7da384c1f38c" containerID="1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5" exitCode=0
Oct 30 03:24:35.917303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.917260    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"
Oct 30 03:24:35.917303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.917274    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh" event={"ID":"090b7c5e-0712-43bf-8205-7da384c1f38c","Type":"ContainerDied","Data":"1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5"}
Oct 30 03:24:35.917515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.917324    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh" event={"ID":"090b7c5e-0712-43bf-8205-7da384c1f38c","Type":"ContainerDied","Data":"de73d3c7cdc3e22e19b05d5b63a1444e99d8a92d345362724c24b31667f609fb"}
Oct 30 03:24:35.917515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.917354    5061 scope.go:122] "RemoveContainer" containerID="1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5"
Oct 30 03:24:35.928592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.928572    5061 scope.go:122] "RemoveContainer" containerID="1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5"
Oct 30 03:24:35.928902 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:24:35.928878    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5\": container with ID starting with 1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5 not found: ID does not exist" containerID="1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5"
Oct 30 03:24:35.928955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.928913    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5"} err="failed to get container status \"1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5\": rpc error: code = NotFound desc = could not find container \"1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5\": container with ID starting with 1ff01bbd12d6637d44753c13d602ff5fe02d614fe63660a82206c35c4371b8b5 not found: ID does not exist"
Oct 30 03:24:35.966805 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.966780    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"]
Oct 30 03:24:35.977130 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:35.977106    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948965-5d4df5d896-tqrqh"]
Oct 30 03:24:36.609647 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:36.609618    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:36.777903 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:36.777861    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:36.943455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:36.943373    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090b7c5e-0712-43bf-8205-7da384c1f38c" path="/var/lib/kubelet/pods/090b7c5e-0712-43bf-8205-7da384c1f38c/volumes"
Oct 30 03:24:37.056906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:37.056872    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:38.179261 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.179221    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs" podUID="e44c0f23-392b-4bee-8c03-82f0b23abe90" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:24:38.704223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.704197    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 03:24:38.816468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.816432    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44c0f23-392b-4bee-8c03-82f0b23abe90-kube-api-access-m7jqf\" (UniqueName: \"kubernetes.io/projected/e44c0f23-392b-4bee-8c03-82f0b23abe90-kube-api-access-m7jqf\") pod \"e44c0f23-392b-4bee-8c03-82f0b23abe90\" (UID: \"e44c0f23-392b-4bee-8c03-82f0b23abe90\") "
Oct 30 03:24:38.816676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.816487    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44c0f23-392b-4bee-8c03-82f0b23abe90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44c0f23-392b-4bee-8c03-82f0b23abe90-plugins-config-volume\") pod \"e44c0f23-392b-4bee-8c03-82f0b23abe90\" (UID: \"e44c0f23-392b-4bee-8c03-82f0b23abe90\") "
Oct 30 03:24:38.817246 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.817220    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44c0f23-392b-4bee-8c03-82f0b23abe90-plugins-config-volume" pod "e44c0f23-392b-4bee-8c03-82f0b23abe90" (UID: "e44c0f23-392b-4bee-8c03-82f0b23abe90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:38.819361 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.819323    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44c0f23-392b-4bee-8c03-82f0b23abe90-kube-api-access-m7jqf" pod "e44c0f23-392b-4bee-8c03-82f0b23abe90" (UID: "e44c0f23-392b-4bee-8c03-82f0b23abe90"). InnerVolumeSpecName "kube-api-access-m7jqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:38.913138 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.913106    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:38.919035 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.919008    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7jqf\" (UniqueName: \"kubernetes.io/projected/e44c0f23-392b-4bee-8c03-82f0b23abe90-kube-api-access-m7jqf\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:38.919170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.919055    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44c0f23-392b-4bee-8c03-82f0b23abe90-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:38.991460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.991420    5061 generic.go:356] "Generic (PLEG): container finished" podID="e44c0f23-392b-4bee-8c03-82f0b23abe90" containerID="56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6" exitCode=0
Oct 30 03:24:38.991590 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.991481    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"
Oct 30 03:24:38.991590 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.991508    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs" event={"ID":"e44c0f23-392b-4bee-8c03-82f0b23abe90","Type":"ContainerDied","Data":"56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6"}
Oct 30 03:24:38.991712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.991604    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs" event={"ID":"e44c0f23-392b-4bee-8c03-82f0b23abe90","Type":"ContainerDied","Data":"37aa62d4aeaa4bfc59244f609b990dd6fdd19d42fb0f58adc4125b3b0a369be9"}
Oct 30 03:24:38.991712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:38.991638    5061 scope.go:122] "RemoveContainer" containerID="56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6"
Oct 30 03:24:39.002249 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:39.002228    5061 scope.go:122] "RemoveContainer" containerID="56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6"
Oct 30 03:24:39.002515 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:24:39.002496    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6\": container with ID starting with 56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6 not found: ID does not exist" containerID="56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6"
Oct 30 03:24:39.002585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:39.002526    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6"} err="failed to get container status \"56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6\": rpc error: code = NotFound desc = could not find container \"56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6\": container with ID starting with 56d75bbc8c57fcba1bee38eb2052c37f970c34da249896eff9eed5afc22ec5a6 not found: ID does not exist"
Oct 30 03:24:39.050540 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:39.050511    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"]
Oct 30 03:24:39.053862 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:39.053824    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-810119-7fc6f9f6cd-dnmhs"]
Oct 30 03:24:40.574018 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:40.573805    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:40.848674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:40.848599    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44c0f23-392b-4bee-8c03-82f0b23abe90" path="/var/lib/kubelet/pods/e44c0f23-392b-4bee-8c03-82f0b23abe90/volumes"
Oct 30 03:24:42.178469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:42.178427    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:42.441630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:42.441553    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:42.573439 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:42.573399    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:42.764783 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:42.764700    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"]
Oct 30 03:24:42.765029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:42.764989    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" podUID="94dba308-852e-4767-a793-5a2e82b13a3e" containerName="istio-proxy" containerID="cri-o://478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d" gracePeriod=30
Oct 30 03:24:42.778309 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:42.778286    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"]
Oct 30 03:24:42.778553 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:42.778513    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk" podUID="a074e7fd-bbf0-4235-ac85-82f9081ec762" containerName="istio-proxy" containerID="cri-o://807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b" gracePeriod=30
Oct 30 03:24:42.780913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:42.780887    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"]
Oct 30 03:24:42.781130 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:42.781094    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy" containerID="cri-o://439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9" gracePeriod=30
Oct 30 03:24:42.798598 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:42.798567    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:43.247307 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:43.247275    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:44.123746 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:44.123709    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:44.584660 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:44.584626    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2" podUID="0dbca485-e20b-4e49-9aa4-3cac4eb994c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:24:44.585061 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:44.584637    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg" podUID="bc979a77-665d-40c3-85f7-66313483e56a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:24:45.016902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.016870    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:24:45.095926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.089580    5061 generic.go:356] "Generic (PLEG): container finished" podID="0dbca485-e20b-4e49-9aa4-3cac4eb994c9" containerID="74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41" exitCode=0
Oct 30 03:24:45.095926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.089747    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2" event={"ID":"0dbca485-e20b-4e49-9aa4-3cac4eb994c9","Type":"ContainerDied","Data":"74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41"}
Oct 30 03:24:45.095926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.089806    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2" event={"ID":"0dbca485-e20b-4e49-9aa4-3cac4eb994c9","Type":"ContainerDied","Data":"fd2858c1b365c6040e65ffc8c9ee4820aae9cfcdac4307aeaeca21baa755c0c8"}
Oct 30 03:24:45.095926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.089837    5061 scope.go:122] "RemoveContainer" containerID="74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41"
Oct 30 03:24:45.134091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.099098    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"
Oct 30 03:24:45.134091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.109433    5061 generic.go:356] "Generic (PLEG): container finished" podID="bc979a77-665d-40c3-85f7-66313483e56a" containerID="0b2dfb54d9662accc60a23910772b4f62aa56fce1a82a09996ad4414777216dc" exitCode=0
Oct 30 03:24:45.134091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.109577    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg" event={"ID":"bc979a77-665d-40c3-85f7-66313483e56a","Type":"ContainerDied","Data":"0b2dfb54d9662accc60a23910772b4f62aa56fce1a82a09996ad4414777216dc"}
Oct 30 03:24:45.134091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.109624    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg" event={"ID":"bc979a77-665d-40c3-85f7-66313483e56a","Type":"ContainerDied","Data":"551bcde1b0de753e7b539227f9d236bc72c7efc811bab980192e5102624661e5"}
Oct 30 03:24:45.134091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.109648    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="551bcde1b0de753e7b539227f9d236bc72c7efc811bab980192e5102624661e5"
Oct 30 03:24:45.166887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.145134    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:24:45.166887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.145489    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-plugins-config-volume\") pod \"0dbca485-e20b-4e49-9aa4-3cac4eb994c9\" (UID: \"0dbca485-e20b-4e49-9aa4-3cac4eb994c9\") "
Oct 30 03:24:45.166887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.145549    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-kube-api-access-tcn6l\" (UniqueName: \"kubernetes.io/projected/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-kube-api-access-tcn6l\") pod \"0dbca485-e20b-4e49-9aa4-3cac4eb994c9\" (UID: \"0dbca485-e20b-4e49-9aa4-3cac4eb994c9\") "
Oct 30 03:24:45.166887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.146084    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-plugins-config-volume" pod "0dbca485-e20b-4e49-9aa4-3cac4eb994c9" (UID: "0dbca485-e20b-4e49-9aa4-3cac4eb994c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:45.166887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.163019    5061 scope.go:122] "RemoveContainer" containerID="74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41"
Oct 30 03:24:45.166887 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:24:45.163581    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41\": container with ID starting with 74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41 not found: ID does not exist" containerID="74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41"
Oct 30 03:24:45.166887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.163640    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41"} err="failed to get container status \"74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41\": rpc error: code = NotFound desc = could not find container \"74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41\": container with ID starting with 74912abe1a3f02e058cbc404d8d10f73e44568c564ef1617e657beb21b692c41 not found: ID does not exist"
Oct 30 03:24:45.166887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.166593    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-kube-api-access-tcn6l" pod "0dbca485-e20b-4e49-9aa4-3cac4eb994c9" (UID: "0dbca485-e20b-4e49-9aa4-3cac4eb994c9"). InnerVolumeSpecName "kube-api-access-tcn6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:45.247502 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.247473    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc979a77-665d-40c3-85f7-66313483e56a-kube-api-access-qr7cr\" (UniqueName: \"kubernetes.io/projected/bc979a77-665d-40c3-85f7-66313483e56a-kube-api-access-qr7cr\") pod \"bc979a77-665d-40c3-85f7-66313483e56a\" (UID: \"bc979a77-665d-40c3-85f7-66313483e56a\") "
Oct 30 03:24:45.247668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.247524    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc979a77-665d-40c3-85f7-66313483e56a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc979a77-665d-40c3-85f7-66313483e56a-plugins-config-volume\") pod \"bc979a77-665d-40c3-85f7-66313483e56a\" (UID: \"bc979a77-665d-40c3-85f7-66313483e56a\") "
Oct 30 03:24:45.247917 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.247891    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc979a77-665d-40c3-85f7-66313483e56a-plugins-config-volume" pod "bc979a77-665d-40c3-85f7-66313483e56a" (UID: "bc979a77-665d-40c3-85f7-66313483e56a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:45.248132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.248119    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc979a77-665d-40c3-85f7-66313483e56a-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:45.248179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.248136    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:45.248179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.248145    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcn6l\" (UniqueName: \"kubernetes.io/projected/0dbca485-e20b-4e49-9aa4-3cac4eb994c9-kube-api-access-tcn6l\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:45.249616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.249593    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc979a77-665d-40c3-85f7-66313483e56a-kube-api-access-qr7cr" pod "bc979a77-665d-40c3-85f7-66313483e56a" (UID: "bc979a77-665d-40c3-85f7-66313483e56a"). InnerVolumeSpecName "kube-api-access-qr7cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:45.349415 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.349344    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr7cr\" (UniqueName: \"kubernetes.io/projected/bc979a77-665d-40c3-85f7-66313483e56a-kube-api-access-qr7cr\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:45.448896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.448866    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"]
Oct 30 03:24:45.458950 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.458923    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-360818-848768dc78-8jws2"]
Oct 30 03:24:45.801785 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.801744    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:45.824588 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.824554    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"]
Oct 30 03:24:45.974115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:45.974024    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:46.119396 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:46.119363    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"
Oct 30 03:24:46.131679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:46.131647    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:46.168587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:46.168562    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"]
Oct 30 03:24:46.178515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:46.178491    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-360818-55b6c65c9f-2vjtg"]
Oct 30 03:24:46.545230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:46.545201    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:46.680970 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:46.680938    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:46.848445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:46.848369    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dbca485-e20b-4e49-9aa4-3cac4eb994c9" path="/var/lib/kubelet/pods/0dbca485-e20b-4e49-9aa4-3cac4eb994c9/volumes"
Oct 30 03:24:46.848827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:46.848811    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc979a77-665d-40c3-85f7-66313483e56a" path="/var/lib/kubelet/pods/bc979a77-665d-40c3-85f7-66313483e56a/volumes"
Oct 30 03:24:47.408955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:47.408915    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:24:47.409145 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:47.408976    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:24:47.461082 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:47.461046    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:47.692183 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:47.692094    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:48.066702 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.066667    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:24:48.099395 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.099360    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-istio-token\" (UniqueName: \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-istio-token\") pod \"94dba308-852e-4767-a793-5a2e82b13a3e\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") "
Oct 30 03:24:48.099543 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.099464    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-credential-socket\") pod \"94dba308-852e-4767-a793-5a2e82b13a3e\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") "
Oct 30 03:24:48.099619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.099577    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94dba308-852e-4767-a793-5a2e82b13a3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94dba308-852e-4767-a793-5a2e82b13a3e-istio-podinfo\") pod \"94dba308-852e-4767-a793-5a2e82b13a3e\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") "
Oct 30 03:24:48.099685 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.099629    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-kube-api-access-6jqff\" (UniqueName: \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-kube-api-access-6jqff\") pod \"94dba308-852e-4767-a793-5a2e82b13a3e\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") "
Oct 30 03:24:48.099685 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.099676    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94dba308-852e-4767-a793-5a2e82b13a3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94dba308-852e-4767-a793-5a2e82b13a3e-istiod-ca-cert\") pod \"94dba308-852e-4767-a793-5a2e82b13a3e\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") "
Oct 30 03:24:48.099797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.099774    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-credential-socket" pod "94dba308-852e-4767-a793-5a2e82b13a3e" (UID: "94dba308-852e-4767-a793-5a2e82b13a3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.099884 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.099832    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-socket\") pod \"94dba308-852e-4767-a793-5a2e82b13a3e\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") "
Oct 30 03:24:48.099954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.099904    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-envoy\") pod \"94dba308-852e-4767-a793-5a2e82b13a3e\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") "
Oct 30 03:24:48.100048 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.100026    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-data\") pod \"94dba308-852e-4767-a793-5a2e82b13a3e\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") "
Oct 30 03:24:48.100167 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.100129    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-certs\") pod \"94dba308-852e-4767-a793-5a2e82b13a3e\" (UID: \"94dba308-852e-4767-a793-5a2e82b13a3e\") "
Oct 30 03:24:48.100265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.100242    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-socket" pod "94dba308-852e-4767-a793-5a2e82b13a3e" (UID: "94dba308-852e-4767-a793-5a2e82b13a3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.101087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.100405    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-data" pod "94dba308-852e-4767-a793-5a2e82b13a3e" (UID: "94dba308-852e-4767-a793-5a2e82b13a3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.101087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.100632    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-certs" pod "94dba308-852e-4767-a793-5a2e82b13a3e" (UID: "94dba308-852e-4767-a793-5a2e82b13a3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.101087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.100695    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94dba308-852e-4767-a793-5a2e82b13a3e-istiod-ca-cert" pod "94dba308-852e-4767-a793-5a2e82b13a3e" (UID: "94dba308-852e-4767-a793-5a2e82b13a3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:48.101087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.100995    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.101087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.101016    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.101087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.101030    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.101087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.101043    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.101087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.101055    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94dba308-852e-4767-a793-5a2e82b13a3e-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.103042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.102218    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-istio-token" pod "94dba308-852e-4767-a793-5a2e82b13a3e" (UID: "94dba308-852e-4767-a793-5a2e82b13a3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.103042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.102838    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-kube-api-access-6jqff" pod "94dba308-852e-4767-a793-5a2e82b13a3e" (UID: "94dba308-852e-4767-a793-5a2e82b13a3e"). InnerVolumeSpecName "kube-api-access-6jqff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.103690 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.103657    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-envoy" pod "94dba308-852e-4767-a793-5a2e82b13a3e" (UID: "94dba308-852e-4767-a793-5a2e82b13a3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.103796 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.103705    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94dba308-852e-4767-a793-5a2e82b13a3e-istio-podinfo" pod "94dba308-852e-4767-a793-5a2e82b13a3e" (UID: "94dba308-852e-4767-a793-5a2e82b13a3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:48.138162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.138133    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:24:48.142979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.142955    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:24:48.197315 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.197049    5061 generic.go:356] "Generic (PLEG): container finished" podID="94dba308-852e-4767-a793-5a2e82b13a3e" containerID="478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d" exitCode=0
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.197678    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.198783    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" event={"ID":"94dba308-852e-4767-a793-5a2e82b13a3e","Type":"ContainerDied","Data":"478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d"}
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.198886    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf" event={"ID":"94dba308-852e-4767-a793-5a2e82b13a3e","Type":"ContainerDied","Data":"becf31d528f20f8f96a2876dfb8954f527b047b96224f864fc7eba1b4574a901"}
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.198932    5061 scope.go:122] "RemoveContainer" containerID="478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d"
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.200564    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"]
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.201200    5061 generic.go:356] "Generic (PLEG): container finished" podID="a074e7fd-bbf0-4235-ac85-82f9081ec762" containerID="807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b" exitCode=0
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.201379    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk" event={"ID":"a074e7fd-bbf0-4235-ac85-82f9081ec762","Type":"ContainerDied","Data":"807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b"}
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.201424    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk" event={"ID":"a074e7fd-bbf0-4235-ac85-82f9081ec762","Type":"ContainerDied","Data":"7a07081f5041ba29ec9473d76f90ae2fc7703af6c0d5d49aaebe4d4537434ef3"}
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.201508    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202220    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3f605d-7d6f-4e51-ada8-12357911adc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3f605d-7d6f-4e51-ada8-12357911adc7-istiod-ca-cert\") pod \"4a3f605d-7d6f-4e51-ada8-12357911adc7\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202315    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-certs\") pod \"4a3f605d-7d6f-4e51-ada8-12357911adc7\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202412    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-socket\") pod \"4a3f605d-7d6f-4e51-ada8-12357911adc7\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202489    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-data\") pod \"a074e7fd-bbf0-4235-ac85-82f9081ec762\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202544    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a074e7fd-bbf0-4235-ac85-82f9081ec762-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a074e7fd-bbf0-4235-ac85-82f9081ec762-istiod-ca-cert\") pod \"a074e7fd-bbf0-4235-ac85-82f9081ec762\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202603    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-certs\") pod \"a074e7fd-bbf0-4235-ac85-82f9081ec762\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202696    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-kube-api-access-ksrk4\" (UniqueName: \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-kube-api-access-ksrk4\") pod \"4a3f605d-7d6f-4e51-ada8-12357911adc7\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202749    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-envoy\") pod \"4a3f605d-7d6f-4e51-ada8-12357911adc7\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202784    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-credential-socket\") pod \"a074e7fd-bbf0-4235-ac85-82f9081ec762\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202839    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-credential-socket\") pod \"4a3f605d-7d6f-4e51-ada8-12357911adc7\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202896    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-token\" (UniqueName: \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-token\") pod \"a074e7fd-bbf0-4235-ac85-82f9081ec762\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202951    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-data\") pod \"4a3f605d-7d6f-4e51-ada8-12357911adc7\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.202997    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-token\" (UniqueName: \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-token\") pod \"4a3f605d-7d6f-4e51-ada8-12357911adc7\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.203116    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-kube-api-access-q52hb\" (UniqueName: \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-kube-api-access-q52hb\") pod \"a074e7fd-bbf0-4235-ac85-82f9081ec762\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.203158    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-podinfo\") pod \"4a3f605d-7d6f-4e51-ada8-12357911adc7\" (UID: \"4a3f605d-7d6f-4e51-ada8-12357911adc7\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.203230    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-podinfo\") pod \"a074e7fd-bbf0-4235-ac85-82f9081ec762\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.203321    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-socket\") pod \"a074e7fd-bbf0-4235-ac85-82f9081ec762\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.203358    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-envoy\") pod \"a074e7fd-bbf0-4235-ac85-82f9081ec762\" (UID: \"a074e7fd-bbf0-4235-ac85-82f9081ec762\") "
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.204101    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.204125    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94dba308-852e-4767-a793-5a2e82b13a3e-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.204140    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jqff\" (UniqueName: \"kubernetes.io/projected/94dba308-852e-4767-a793-5a2e82b13a3e-kube-api-access-6jqff\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.204156    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94dba308-852e-4767-a793-5a2e82b13a3e-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.204363    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-data" pod "a074e7fd-bbf0-4235-ac85-82f9081ec762" (UID: "a074e7fd-bbf0-4235-ac85-82f9081ec762"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.204583    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-credential-socket" pod "a074e7fd-bbf0-4235-ac85-82f9081ec762" (UID: "a074e7fd-bbf0-4235-ac85-82f9081ec762"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.206100    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3f605d-7d6f-4e51-ada8-12357911adc7-istiod-ca-cert" pod "4a3f605d-7d6f-4e51-ada8-12357911adc7" (UID: "4a3f605d-7d6f-4e51-ada8-12357911adc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.206261    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a074e7fd-bbf0-4235-ac85-82f9081ec762-istiod-ca-cert" pod "a074e7fd-bbf0-4235-ac85-82f9081ec762" (UID: "a074e7fd-bbf0-4235-ac85-82f9081ec762"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.207585    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-credential-socket" pod "4a3f605d-7d6f-4e51-ada8-12357911adc7" (UID: "4a3f605d-7d6f-4e51-ada8-12357911adc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.209789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.207654    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-socket" pod "4a3f605d-7d6f-4e51-ada8-12357911adc7" (UID: "4a3f605d-7d6f-4e51-ada8-12357911adc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.215328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.215107    5061 generic.go:356] "Generic (PLEG): container finished" podID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerID="439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9" exitCode=0
Oct 30 03:24:48.215328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.215152    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" event={"ID":"4a3f605d-7d6f-4e51-ada8-12357911adc7","Type":"ContainerDied","Data":"439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9"}
Oct 30 03:24:48.215328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.215190    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn" event={"ID":"4a3f605d-7d6f-4e51-ada8-12357911adc7","Type":"ContainerDied","Data":"c643caddb9b0e76643f3827e27b83017e8e85947bdcbe1c82e554f5ffc16c42e"}
Oct 30 03:24:48.215328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.215266    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"
Oct 30 03:24:48.216717 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.216307    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-certs" pod "a074e7fd-bbf0-4235-ac85-82f9081ec762" (UID: "a074e7fd-bbf0-4235-ac85-82f9081ec762"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.218419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.218390    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-podinfo" pod "a074e7fd-bbf0-4235-ac85-82f9081ec762" (UID: "a074e7fd-bbf0-4235-ac85-82f9081ec762"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:48.218527 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.218450    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-socket" pod "a074e7fd-bbf0-4235-ac85-82f9081ec762" (UID: "a074e7fd-bbf0-4235-ac85-82f9081ec762"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.218527 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.218513    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-certs" pod "4a3f605d-7d6f-4e51-ada8-12357911adc7" (UID: "4a3f605d-7d6f-4e51-ada8-12357911adc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.218745 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.218682    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-data" pod "4a3f605d-7d6f-4e51-ada8-12357911adc7" (UID: "4a3f605d-7d6f-4e51-ada8-12357911adc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.221958 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.221936    5061 scope.go:122] "RemoveContainer" containerID="478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d"
Oct 30 03:24:48.223097 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:24:48.223069    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d\": container with ID starting with 478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d not found: ID does not exist" containerID="478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d"
Oct 30 03:24:48.223198 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.223110    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d"} err="failed to get container status \"478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d\": rpc error: code = NotFound desc = could not find container \"478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d\": container with ID starting with 478e3cf1bcbdcf10fad9e8748274ac7412da7cad04e7df713c2707ab5928a75d not found: ID does not exist"
Oct 30 03:24:48.223198 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.223133    5061 scope.go:122] "RemoveContainer" containerID="807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b"
Oct 30 03:24:48.229569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.227130    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-token" pod "4a3f605d-7d6f-4e51-ada8-12357911adc7" (UID: "4a3f605d-7d6f-4e51-ada8-12357911adc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.231024 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.230964    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-kube-api-access-q52hb" pod "a074e7fd-bbf0-4235-ac85-82f9081ec762" (UID: "a074e7fd-bbf0-4235-ac85-82f9081ec762"). InnerVolumeSpecName "kube-api-access-q52hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.231169 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.231055    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-kube-api-access-ksrk4" pod "4a3f605d-7d6f-4e51-ada8-12357911adc7" (UID: "4a3f605d-7d6f-4e51-ada8-12357911adc7"). InnerVolumeSpecName "kube-api-access-ksrk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.231169 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.231060    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-podinfo" pod "4a3f605d-7d6f-4e51-ada8-12357911adc7" (UID: "4a3f605d-7d6f-4e51-ada8-12357911adc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:48.231301 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.231260    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-envoy" pod "4a3f605d-7d6f-4e51-ada8-12357911adc7" (UID: "4a3f605d-7d6f-4e51-ada8-12357911adc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.231495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.231473    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-envoy" pod "a074e7fd-bbf0-4235-ac85-82f9081ec762" (UID: "a074e7fd-bbf0-4235-ac85-82f9081ec762"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:48.233662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.233635    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-token" pod "a074e7fd-bbf0-4235-ac85-82f9081ec762" (UID: "a074e7fd-bbf0-4235-ac85-82f9081ec762"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:48.241664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.241647    5061 scope.go:122] "RemoveContainer" containerID="807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b"
Oct 30 03:24:48.242017 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:24:48.241995    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b\": container with ID starting with 807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b not found: ID does not exist" containerID="807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b"
Oct 30 03:24:48.242106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.242023    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b"} err="failed to get container status \"807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b\": rpc error: code = NotFound desc = could not find container \"807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b\": container with ID starting with 807d9d4406128a8421ad23ea23e1a60a34c0c4cb738552ea3837ff9084ab979b not found: ID does not exist"
Oct 30 03:24:48.242106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.242039    5061 scope.go:122] "RemoveContainer" containerID="439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9"
Oct 30 03:24:48.253653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.253630    5061 scope.go:122] "RemoveContainer" containerID="c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265"
Oct 30 03:24:48.266125 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.266106    5061 scope.go:122] "RemoveContainer" containerID="439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9"
Oct 30 03:24:48.266827 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:24:48.266799    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9\": container with ID starting with 439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9 not found: ID does not exist" containerID="439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9"
Oct 30 03:24:48.266940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.266834    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9"} err="failed to get container status \"439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9\": rpc error: code = NotFound desc = could not find container \"439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9\": container with ID starting with 439c0616311cf160be3dc1fd037bdb89d63763e9285a29ffbb90b5ef740617a9 not found: ID does not exist"
Oct 30 03:24:48.266940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.266898    5061 scope.go:122] "RemoveContainer" containerID="c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265"
Oct 30 03:24:48.267220 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:24:48.267195    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265\": container with ID starting with c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265 not found: ID does not exist" containerID="c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265"
Oct 30 03:24:48.267313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.267228    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265"} err="failed to get container status \"c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265\": rpc error: code = NotFound desc = could not find container \"c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265\": container with ID starting with c80ce919cc9f9aad66ceed827971a3b2f007bc227e5d153e6f779ec3f49c1265 not found: ID does not exist"
Oct 30 03:24:48.304969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.304945    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.304969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.304968    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.304979    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a074e7fd-bbf0-4235-ac85-82f9081ec762-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.304988    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.304996    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksrk4\" (UniqueName: \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-kube-api-access-ksrk4\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305005    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305013    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305030    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305039    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305048    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305056    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305065    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q52hb\" (UniqueName: \"kubernetes.io/projected/a074e7fd-bbf0-4235-ac85-82f9081ec762-kube-api-access-q52hb\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305072    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3f605d-7d6f-4e51-ada8-12357911adc7-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305080    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305088    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305096    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a074e7fd-bbf0-4235-ac85-82f9081ec762-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305114    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3f605d-7d6f-4e51-ada8-12357911adc7-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.305495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.305122    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3f605d-7d6f-4e51-ada8-12357911adc7-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:24:48.313709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.313679    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:48.338292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.338256    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"]
Oct 30 03:24:48.341651 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.341629    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-2ljdf"]
Oct 30 03:24:48.545117 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.545089    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"]
Oct 30 03:24:48.548816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.548783    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-69ckk"]
Oct 30 03:24:48.580353 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.580327    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"]
Oct 30 03:24:48.584133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.584113    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-360818-openshift-default-65c5f7bf4f-4q4vn"]
Oct 30 03:24:48.849509 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.849476    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" path="/var/lib/kubelet/pods/4a3f605d-7d6f-4e51-ada8-12357911adc7/volumes"
Oct 30 03:24:48.850028 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.850011    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94dba308-852e-4767-a793-5a2e82b13a3e" path="/var/lib/kubelet/pods/94dba308-852e-4767-a793-5a2e82b13a3e/volumes"
Oct 30 03:24:48.850469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:48.850455    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a074e7fd-bbf0-4235-ac85-82f9081ec762" path="/var/lib/kubelet/pods/a074e7fd-bbf0-4235-ac85-82f9081ec762/volumes"
Oct 30 03:24:49.005369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.005336    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:49.130094 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.130013    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"]
Oct 30 03:24:49.130976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.130958    5061 cpu_manager.go:414] "containerMap: removing container" podUID="a074e7fd-bbf0-4235-ac85-82f9081ec762" containerName="istio-proxy"
Oct 30 03:24:49.131039 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.130979    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a074e7fd-bbf0-4235-ac85-82f9081ec762" containerName="istio-proxy"
Oct 30 03:24:49.131039 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.130992    5061 cpu_manager.go:414] "containerMap: removing container" podUID="090b7c5e-0712-43bf-8205-7da384c1f38c" containerName="payload-processing"
Oct 30 03:24:49.131039 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131000    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090b7c5e-0712-43bf-8205-7da384c1f38c" containerName="payload-processing"
Oct 30 03:24:49.131039 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131026    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main"
Oct 30 03:24:49.131039 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131034    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131053    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0dbca485-e20b-4e49-9aa4-3cac4eb994c9" containerName="payload-processing"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131061    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbca485-e20b-4e49-9aa4-3cac4eb994c9" containerName="payload-processing"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131081    5061 cpu_manager.go:414] "containerMap: removing container" podUID="94dba308-852e-4767-a793-5a2e82b13a3e" containerName="istio-proxy"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131089    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94dba308-852e-4767-a793-5a2e82b13a3e" containerName="istio-proxy"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131101    5061 cpu_manager.go:414] "containerMap: removing container" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerName="istio-proxy"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131108    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerName="istio-proxy"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131125    5061 cpu_manager.go:414] "containerMap: removing container" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerName="istio-proxy"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131132    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerName="istio-proxy"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131143    5061 cpu_manager.go:414] "containerMap: removing container" podUID="bc979a77-665d-40c3-85f7-66313483e56a" containerName="payload-pre-processing"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131151    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc979a77-665d-40c3-85f7-66313483e56a" containerName="payload-pre-processing"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131165    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy"
Oct 30 03:24:49.131193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131171    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131201    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e44c0f23-392b-4bee-8c03-82f0b23abe90" containerName="payload-processing"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131210    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44c0f23-392b-4bee-8c03-82f0b23abe90" containerName="payload-processing"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131225    5061 cpu_manager.go:414] "containerMap: removing container" podUID="af43e5ce-2da5-4f93-9296-66e75df36e82" containerName="istio-proxy"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131232    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af43e5ce-2da5-4f93-9296-66e75df36e82" containerName="istio-proxy"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131242    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131249    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131405    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44c0f23-392b-4bee-8c03-82f0b23abe90" containerName="payload-processing"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131419    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="af43e5ce-2da5-4f93-9296-66e75df36e82" containerName="istio-proxy"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131428    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131439    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="a074e7fd-bbf0-4235-ac85-82f9081ec762" containerName="istio-proxy"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131447    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="090b7c5e-0712-43bf-8205-7da384c1f38c" containerName="payload-processing"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131459    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e076698c-e0eb-42d9-bdd9-2ecd3318da04" containerName="main"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131470    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbca485-e20b-4e49-9aa4-3cac4eb994c9" containerName="payload-processing"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131480    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="94dba308-852e-4767-a793-5a2e82b13a3e" containerName="istio-proxy"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131489    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bd36df-8df5-4293-9a2a-efa4da29d75d" containerName="istio-proxy"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131499    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c3171d-e418-4df4-b4dc-aa9a8ced8706" containerName="istio-proxy"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131507    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc979a77-665d-40c3-85f7-66313483e56a" containerName="payload-pre-processing"
Oct 30 03:24:49.131537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.131517    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3f605d-7d6f-4e51-ada8-12357911adc7" containerName="istio-proxy"
Oct 30 03:24:49.136825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.136804    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.155637 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.155615    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-d823d1-openshift-default-dockercfg-bhzs8\""
Oct 30 03:24:49.159457 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.159431    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:49.160359 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.160328    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"]
Oct 30 03:24:49.219567 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.219529    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-istio-token\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.219726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.219582    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-credential-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.219726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.219674    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-envoy\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.219803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.219757    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-certs\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.219866 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.219836    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-data\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.219944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.219930    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7fd568-5977-42fc-987d-61af621d4d38-istio-podinfo\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.220081 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.220063    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss79f\" (UniqueName: \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-kube-api-access-ss79f\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.220151 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.220138    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.220245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.220234    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7fd568-5977-42fc-987d-61af621d4d38-istiod-ca-cert\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.321401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.321365    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7fd568-5977-42fc-987d-61af621d4d38-istio-podinfo\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.321575 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.321440    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss79f\" (UniqueName: \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-kube-api-access-ss79f\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.321575 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.321501    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.321575 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.321554    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7fd568-5977-42fc-987d-61af621d4d38-istiod-ca-cert\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.321679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.321583    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-istio-token\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.321679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.321627    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-credential-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.321679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.321649    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-envoy\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.321775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.321681    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-certs\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.321775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.321707    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-data\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.322111 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.322086    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-data\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.323137 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.323109    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.323477 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.323449    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-credential-socket\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.323691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.323668    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-certs\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.323775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.323760    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7fd568-5977-42fc-987d-61af621d4d38-istiod-ca-cert\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.324666 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.324641    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7fd568-5977-42fc-987d-61af621d4d38-istio-podinfo\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.325129 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.325108    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-envoy\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.346960 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.346932    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-istio-token\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.355002 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.354970    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss79f\" (UniqueName: \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-kube-api-access-ss79f\") pod \"e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") " pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.451586 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.451500    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:49.781152 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.781071    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"]
Oct 30 03:24:49.785129 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:24:49.785102    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7fd568_5977_42fc_987d_61af621d4d38.slice/crio-9ed4e368aefc1b3e7d0c2366ae870bef5ab65e4770c3a6a2d1b33f02762c2afb WatchSource:0}: Error finding container 9ed4e368aefc1b3e7d0c2366ae870bef5ab65e4770c3a6a2d1b33f02762c2afb: Status 404 returned error can't find the container with id 9ed4e368aefc1b3e7d0c2366ae870bef5ab65e4770c3a6a2d1b33f02762c2afb
Oct 30 03:24:49.789362 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.789304    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:24:49.789492 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.789471    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:24:49.789580 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:49.789559    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:24:50.101461 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:50.101423    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:50.235781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:50.235744    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" event={"ID":"2a7fd568-5977-42fc-987d-61af621d4d38","Type":"ContainerStarted","Data":"626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874"}
Oct 30 03:24:50.236172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:50.235826    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" event={"ID":"2a7fd568-5977-42fc-987d-61af621d4d38","Type":"ContainerStarted","Data":"9ed4e368aefc1b3e7d0c2366ae870bef5ab65e4770c3a6a2d1b33f02762c2afb"}
Oct 30 03:24:50.301639 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:50.301574    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" podStartSLOduration=1.3015561820000001 podStartE2EDuration="1.301556182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:50.298551113 +0000 UTC m=+2635.600729477" watchObservedRunningTime="2026-10-30 03:24:50.301556182 +0000 UTC m=+2635.603734548"
Oct 30 03:24:50.452240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:50.452141    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:50.454383 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:50.454355    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:24:50.454524 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:50.454407    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" podUID="2a7fd568-5977-42fc-987d-61af621d4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 03:24:51.172490 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:51.172138    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:51.372927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:51.372890    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:51.452445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:51.452355    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:24:51.452445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:51.452423    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" podUID="2a7fd568-5977-42fc-987d-61af621d4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 03:24:52.453204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:52.453162    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:24:52.453628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:52.453221    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" podUID="2a7fd568-5977-42fc-987d-61af621d4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 03:24:52.795423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:52.795385    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:53.045887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:53.045733    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:53.197808 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:53.197776    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:53.411932 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:53.399172    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:53.462875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:53.462640    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:24:53.462875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:53.462698    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" podUID="2a7fd568-5977-42fc-987d-61af621d4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 03:24:53.844825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:53.844777    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:53.989512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:53.989477    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:54.458087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:54.458038    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:24:54.458445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:54.458395    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" podUID="2a7fd568-5977-42fc-987d-61af621d4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 03:24:55.382796 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:55.382756    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:55.452874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:55.452345    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:24:55.452874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:55.452414    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" podUID="2a7fd568-5977-42fc-987d-61af621d4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 03:24:55.489924 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:24:55.489839    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:24:56.462320 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:56.462284    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:56.463100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:56.463076    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:56.464414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:56.464384    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:24:57.001491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:57.001462    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:57.046579 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:57.046543    5061 scope.go:122] "RemoveContainer" containerID="0b2dfb54d9662accc60a23910772b4f62aa56fce1a82a09996ad4414777216dc"
Oct 30 03:24:58.588817 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:24:58.588756    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:00.305328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:00.305291    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:00.845500 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:00.845468    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99" podUID="8bf611f0-2273-4646-8cfc-bd12ec8a2dee" containerName="main" containerID="cri-o://805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485" gracePeriod=45
Oct 30 03:25:01.110955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.110893    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:25:01.277577 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.277549    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-kube-api-access-pq7mb\" (UniqueName: \"kubernetes.io/projected/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-kube-api-access-pq7mb\") pod \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") "
Oct 30 03:25:01.277743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.277599    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tmp-dir\") pod \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") "
Oct 30 03:25:01.277743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.277631    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-home\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-home\") pod \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") "
Oct 30 03:25:01.277875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.277748    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-dshm\") pod \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") "
Oct 30 03:25:01.277875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.277790    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-model-cache\") pod \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") "
Oct 30 03:25:01.278012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.277884    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tmp-dir" pod "8bf611f0-2273-4646-8cfc-bd12ec8a2dee" (UID: "8bf611f0-2273-4646-8cfc-bd12ec8a2dee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:01.278012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.277916    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tls-certs\") pod \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\" (UID: \"8bf611f0-2273-4646-8cfc-bd12ec8a2dee\") "
Oct 30 03:25:01.278012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.277936    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-home" pod "8bf611f0-2273-4646-8cfc-bd12ec8a2dee" (UID: "8bf611f0-2273-4646-8cfc-bd12ec8a2dee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:01.278181 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.278128    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-model-cache" pod "8bf611f0-2273-4646-8cfc-bd12ec8a2dee" (UID: "8bf611f0-2273-4646-8cfc-bd12ec8a2dee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:01.278459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.278438    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:01.278459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.278457    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:01.278459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.278466    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:01.280055 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.280030    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-dshm" pod "8bf611f0-2273-4646-8cfc-bd12ec8a2dee" (UID: "8bf611f0-2273-4646-8cfc-bd12ec8a2dee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:01.280158 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.280086    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tls-certs" pod "8bf611f0-2273-4646-8cfc-bd12ec8a2dee" (UID: "8bf611f0-2273-4646-8cfc-bd12ec8a2dee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:01.280158 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.280132    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-kube-api-access-pq7mb" pod "8bf611f0-2273-4646-8cfc-bd12ec8a2dee" (UID: "8bf611f0-2273-4646-8cfc-bd12ec8a2dee"). InnerVolumeSpecName "kube-api-access-pq7mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:01.328729 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.328702    5061 generic.go:356] "Generic (PLEG): container finished" podID="8bf611f0-2273-4646-8cfc-bd12ec8a2dee" containerID="805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485" exitCode=0
Oct 30 03:25:01.329138 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.328749    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"
Oct 30 03:25:01.329138 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.328785    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99" event={"ID":"8bf611f0-2273-4646-8cfc-bd12ec8a2dee","Type":"ContainerDied","Data":"805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485"}
Oct 30 03:25:01.329138 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.328882    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99" event={"ID":"8bf611f0-2273-4646-8cfc-bd12ec8a2dee","Type":"ContainerDied","Data":"a94d0393a80757d4f40c17d77462a5b03a73a0a5f92b9cac56aa03b8fd5dd1e7"}
Oct 30 03:25:01.329138 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.328910    5061 scope.go:122] "RemoveContainer" containerID="805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485"
Oct 30 03:25:01.339185 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.339167    5061 scope.go:122] "RemoveContainer" containerID="805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485"
Oct 30 03:25:01.339460 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:25:01.339441    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485\": container with ID starting with 805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485 not found: ID does not exist" containerID="805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485"
Oct 30 03:25:01.339500 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.339469    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485"} err="failed to get container status \"805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485\": rpc error: code = NotFound desc = could not find container \"805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485\": container with ID starting with 805a604c412beca5e9a6e3c16ea3b05a15ac4d5c14ba39cf3b5cfeb83bb8d485 not found: ID does not exist"
Oct 30 03:25:01.380058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.379963    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq7mb\" (UniqueName: \"kubernetes.io/projected/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-kube-api-access-pq7mb\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:01.380058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.379999    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:01.380058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.380014    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf611f0-2273-4646-8cfc-bd12ec8a2dee-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:01.381821 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.381789    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"]
Oct 30 03:25:01.386590 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.386554    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-ae1dd9/auth-test-model-ae1dd9-kserve-76f5d5f6df-5ck99"]
Oct 30 03:25:01.872007 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:01.871949    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:02.849768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:02.849731    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf611f0-2273-4646-8cfc-bd12ec8a2dee" path="/var/lib/kubelet/pods/8bf611f0-2273-4646-8cfc-bd12ec8a2dee/volumes"
Oct 30 03:25:03.247266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.247182    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" containerID="cri-o://733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb" gracePeriod=45
Oct 30 03:25:03.403536 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.403495    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 03:25:03.420021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.419989    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:03.554757 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.554728    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 03:25:03.712156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.712123    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-dshm\") pod \"1318fb59-a096-440a-b3cb-43139a3bc387\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") "
Oct 30 03:25:03.712356 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.712244    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-tmp-dir\") pod \"1318fb59-a096-440a-b3cb-43139a3bc387\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") "
Oct 30 03:25:03.712439 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.712357    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1318fb59-a096-440a-b3cb-43139a3bc387-kube-api-access-b9zjd\" (UniqueName: \"kubernetes.io/projected/1318fb59-a096-440a-b3cb-43139a3bc387-kube-api-access-b9zjd\") pod \"1318fb59-a096-440a-b3cb-43139a3bc387\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") "
Oct 30 03:25:03.712439 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.712378    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-home\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-home\") pod \"1318fb59-a096-440a-b3cb-43139a3bc387\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") "
Oct 30 03:25:03.712439 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.712415    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-model-cache\") pod \"1318fb59-a096-440a-b3cb-43139a3bc387\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") "
Oct 30 03:25:03.712605 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.712463    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1318fb59-a096-440a-b3cb-43139a3bc387-tls-certs\" (UniqueName: \"kubernetes.io/secret/1318fb59-a096-440a-b3cb-43139a3bc387-tls-certs\") pod \"1318fb59-a096-440a-b3cb-43139a3bc387\" (UID: \"1318fb59-a096-440a-b3cb-43139a3bc387\") "
Oct 30 03:25:03.712605 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.712535    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-tmp-dir" pod "1318fb59-a096-440a-b3cb-43139a3bc387" (UID: "1318fb59-a096-440a-b3cb-43139a3bc387"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:03.712725 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.712638    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-home" pod "1318fb59-a096-440a-b3cb-43139a3bc387" (UID: "1318fb59-a096-440a-b3cb-43139a3bc387"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:03.712782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.712751    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-model-cache" pod "1318fb59-a096-440a-b3cb-43139a3bc387" (UID: "1318fb59-a096-440a-b3cb-43139a3bc387"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:03.713205 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.713178    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:03.713314 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.713230    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:03.713314 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.713245    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:03.714660 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.714631    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-dshm" pod "1318fb59-a096-440a-b3cb-43139a3bc387" (UID: "1318fb59-a096-440a-b3cb-43139a3bc387"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:03.714769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.714656    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1318fb59-a096-440a-b3cb-43139a3bc387-tls-certs" pod "1318fb59-a096-440a-b3cb-43139a3bc387" (UID: "1318fb59-a096-440a-b3cb-43139a3bc387"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:03.714815 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.714780    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1318fb59-a096-440a-b3cb-43139a3bc387-kube-api-access-b9zjd" pod "1318fb59-a096-440a-b3cb-43139a3bc387" (UID: "1318fb59-a096-440a-b3cb-43139a3bc387"). InnerVolumeSpecName "kube-api-access-b9zjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:03.814955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.814923    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1318fb59-a096-440a-b3cb-43139a3bc387-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:03.814955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.814953    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1318fb59-a096-440a-b3cb-43139a3bc387-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:03.815166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:03.814966    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9zjd\" (UniqueName: \"kubernetes.io/projected/1318fb59-a096-440a-b3cb-43139a3bc387-kube-api-access-b9zjd\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:04.350719 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.350685    5061 generic.go:356] "Generic (PLEG): container finished" podID="1318fb59-a096-440a-b3cb-43139a3bc387" containerID="733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb" exitCode=0
Oct 30 03:25:04.351149 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.350746    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"
Oct 30 03:25:04.351149 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.350769    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" event={"ID":"1318fb59-a096-440a-b3cb-43139a3bc387","Type":"ContainerDied","Data":"733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb"}
Oct 30 03:25:04.351149 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.350815    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5" event={"ID":"1318fb59-a096-440a-b3cb-43139a3bc387","Type":"ContainerDied","Data":"a581535c5e8453701353d4d60075473b57504ea94cfb84df9c099f4312389529"}
Oct 30 03:25:04.351149 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.350835    5061 scope.go:122] "RemoveContainer" containerID="733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb"
Oct 30 03:25:04.361677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.361654    5061 scope.go:122] "RemoveContainer" containerID="733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb"
Oct 30 03:25:04.362068 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:25:04.362046    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb\": container with ID starting with 733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb not found: ID does not exist" containerID="733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb"
Oct 30 03:25:04.362136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.362075    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb"} err="failed to get container status \"733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb\": rpc error: code = NotFound desc = could not find container \"733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb\": container with ID starting with 733dccf9709d8a3fd58c73e2f07a6d7c9620b8b9f66532ff5aef73bc1d84cddb not found: ID does not exist"
Oct 30 03:25:04.399091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.399056    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"]
Oct 30 03:25:04.409101 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.409072    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7d1788/distinct-w4-7d1788-kserve-67df45cb4b-n2ch5"]
Oct 30 03:25:04.627017 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.626939    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:04.746178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.746136    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:04.848710 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:04.848675    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" path="/var/lib/kubelet/pods/1318fb59-a096-440a-b3cb-43139a3bc387/volumes"
Oct 30 03:25:05.975192 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:05.975118    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:07.525030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:07.524998    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:09.074656 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:09.074616    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:10.595885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:10.595836    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:10.666958 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:10.666922    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"]
Oct 30 03:25:10.667164 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:10.667125    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv" podUID="c24ea1dc-460b-4d52-83d5-17c97ca3e29b" containerName="payload-processing" containerID="cri-o://aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841" gracePeriod=30
Oct 30 03:25:10.694890 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:10.694863    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"]
Oct 30 03:25:10.695091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:10.695070    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp" podUID="c12a8d12-5c1d-492c-a468-a65a8f06fadf" containerName="payload-pre-processing" containerID="cri-o://aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206" gracePeriod=30
Oct 30 03:25:10.768835 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:10.768802    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:10.903360 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:10.903325    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:10.994567 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:10.994542    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:25:11.053598 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.052545    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-plugins-config-volume\") pod \"c24ea1dc-460b-4d52-83d5-17c97ca3e29b\" (UID: \"c24ea1dc-460b-4d52-83d5-17c97ca3e29b\") "
Oct 30 03:25:11.053598 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.052697    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-kube-api-access-ccngf\" (UniqueName: \"kubernetes.io/projected/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-kube-api-access-ccngf\") pod \"c24ea1dc-460b-4d52-83d5-17c97ca3e29b\" (UID: \"c24ea1dc-460b-4d52-83d5-17c97ca3e29b\") "
Oct 30 03:25:11.083869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.073799    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-plugins-config-volume" pod "c24ea1dc-460b-4d52-83d5-17c97ca3e29b" (UID: "c24ea1dc-460b-4d52-83d5-17c97ca3e29b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:11.091978 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.088742    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-kube-api-access-ccngf" pod "c24ea1dc-460b-4d52-83d5-17c97ca3e29b" (UID: "c24ea1dc-460b-4d52-83d5-17c97ca3e29b"). InnerVolumeSpecName "kube-api-access-ccngf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:11.117874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.115803    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:25:11.127150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.125953    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:11.165891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.154541    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12a8d12-5c1d-492c-a468-a65a8f06fadf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12a8d12-5c1d-492c-a468-a65a8f06fadf-plugins-config-volume\") pod \"c12a8d12-5c1d-492c-a468-a65a8f06fadf\" (UID: \"c12a8d12-5c1d-492c-a468-a65a8f06fadf\") "
Oct 30 03:25:11.165891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.154689    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12a8d12-5c1d-492c-a468-a65a8f06fadf-kube-api-access-vznzj\" (UniqueName: \"kubernetes.io/projected/c12a8d12-5c1d-492c-a468-a65a8f06fadf-kube-api-access-vznzj\") pod \"c12a8d12-5c1d-492c-a468-a65a8f06fadf\" (UID: \"c12a8d12-5c1d-492c-a468-a65a8f06fadf\") "
Oct 30 03:25:11.165891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.155252    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:11.165891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.155272    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccngf\" (UniqueName: \"kubernetes.io/projected/c24ea1dc-460b-4d52-83d5-17c97ca3e29b-kube-api-access-ccngf\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:11.165891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.158107    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12a8d12-5c1d-492c-a468-a65a8f06fadf-kube-api-access-vznzj" pod "c12a8d12-5c1d-492c-a468-a65a8f06fadf" (UID: "c12a8d12-5c1d-492c-a468-a65a8f06fadf"). InnerVolumeSpecName "kube-api-access-vznzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:11.165891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.158440    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12a8d12-5c1d-492c-a468-a65a8f06fadf-plugins-config-volume" pod "c12a8d12-5c1d-492c-a468-a65a8f06fadf" (UID: "c12a8d12-5c1d-492c-a468-a65a8f06fadf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:11.256715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.256685    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12a8d12-5c1d-492c-a468-a65a8f06fadf-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:11.256715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.256712    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vznzj\" (UniqueName: \"kubernetes.io/projected/c12a8d12-5c1d-492c-a468-a65a8f06fadf-kube-api-access-vznzj\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:11.361103 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.361020    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:11.408306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.408227    5061 generic.go:356] "Generic (PLEG): container finished" podID="c24ea1dc-460b-4d52-83d5-17c97ca3e29b" containerID="aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841" exitCode=0
Oct 30 03:25:11.408306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.408272    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"
Oct 30 03:25:11.408541 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.408307    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv" event={"ID":"c24ea1dc-460b-4d52-83d5-17c97ca3e29b","Type":"ContainerDied","Data":"aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841"}
Oct 30 03:25:11.408541 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.408366    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv" event={"ID":"c24ea1dc-460b-4d52-83d5-17c97ca3e29b","Type":"ContainerDied","Data":"7887add4ef09b87b38669a05b2b8412e109a0db3be2be10fb4d5479a6548f911"}
Oct 30 03:25:11.408541 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.408403    5061 scope.go:122] "RemoveContainer" containerID="aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841"
Oct 30 03:25:11.409841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.409817    5061 generic.go:356] "Generic (PLEG): container finished" podID="c12a8d12-5c1d-492c-a468-a65a8f06fadf" containerID="aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206" exitCode=0
Oct 30 03:25:11.409975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.409879    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp" event={"ID":"c12a8d12-5c1d-492c-a468-a65a8f06fadf","Type":"ContainerDied","Data":"aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206"}
Oct 30 03:25:11.409975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.409891    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"
Oct 30 03:25:11.409975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.409933    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp" event={"ID":"c12a8d12-5c1d-492c-a468-a65a8f06fadf","Type":"ContainerDied","Data":"93bd0c1ba05134555f629760a0059ba3ad9836eddec94996d32859716533c8b6"}
Oct 30 03:25:11.422695 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.421454    5061 scope.go:122] "RemoveContainer" containerID="aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841"
Oct 30 03:25:11.426623 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:25:11.426596    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841\": container with ID starting with aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841 not found: ID does not exist" containerID="aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841"
Oct 30 03:25:11.426754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.426629    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841"} err="failed to get container status \"aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841\": rpc error: code = NotFound desc = could not find container \"aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841\": container with ID starting with aaf6094dc098d2fbab9875755a099accc3663aee4615cd9831f56dfd1e18c841 not found: ID does not exist"
Oct 30 03:25:11.426754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.426652    5061 scope.go:122] "RemoveContainer" containerID="aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206"
Oct 30 03:25:11.440607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.440582    5061 scope.go:122] "RemoveContainer" containerID="aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206"
Oct 30 03:25:11.441134 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:25:11.441111    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206\": container with ID starting with aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206 not found: ID does not exist" containerID="aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206"
Oct 30 03:25:11.441220 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.441139    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206"} err="failed to get container status \"aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206\": rpc error: code = NotFound desc = could not find container \"aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206\": container with ID starting with aa90656b6472bc3162c61ed9c0b7507d6a2bfbea29d0d04e1da15e02e7fef206 not found: ID does not exist"
Oct 30 03:25:11.461601 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.461580    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"]
Oct 30 03:25:11.465524 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.465502    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-086aee-78d559cdfb-lppwv"]
Oct 30 03:25:11.493586 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.493563    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:11.507587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.507567    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"]
Oct 30 03:25:11.517451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.517429    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-086aee-78d7546f8c-x7shp"]
Oct 30 03:25:11.611341 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.611313    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:11.853229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:11.853197    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:12.001733 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:12.001630    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:12.848900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:12.848863    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12a8d12-5c1d-492c-a468-a65a8f06fadf" path="/var/lib/kubelet/pods/c12a8d12-5c1d-492c-a468-a65a8f06fadf/volumes"
Oct 30 03:25:12.849379 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:12.849363    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24ea1dc-460b-4d52-83d5-17c97ca3e29b" path="/var/lib/kubelet/pods/c24ea1dc-460b-4d52-83d5-17c97ca3e29b/volumes"
Oct 30 03:25:13.052635 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:13.052599    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:13.527562 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:13.527526    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:15.393127 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:15.393094    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:17.501520 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:17.501486    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:18.931820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:18.931785    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:19.498778 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:19.498738    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:19.639372 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:19.639338    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:19.783639 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:19.783552    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:20.112108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:20.112074    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:20.450062 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:20.449976    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:20.646274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:20.646225    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:21.861468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:21.861436    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:24.110015 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:24.109979    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:24.536075 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:24.535955    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:24.726104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:24.726075    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:25.462149 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:25:25.462109    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:25:26.342170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:26.342132    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:26.458811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:26.458780    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:26.661824 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:26.661744    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-v8fgz_87148799-07e3-41d4-a6f0-0e4678d90202/payload-processing/0.log"
Oct 30 03:25:26.873982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:26.873955    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-v8fgz_87148799-07e3-41d4-a6f0-0e4678d90202/payload-processing/0.log"
Oct 30 03:25:27.223657 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:27.223609    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:27.369297 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:27.369262    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:27.394337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:27.394301    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"]
Oct 30 03:25:27.394581 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:27.394552    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy" containerID="cri-o://e762852e18318669d25b77e591ed42b08dbead4cce0921e7d4003e845638293c" gracePeriod=30
Oct 30 03:25:27.540937 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:27.540137    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:28.179577 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:28.179536    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:28.309643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:28.309565    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:30.249986 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:30.249951    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:31.887309 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:31.887271    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:32.659463 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.659306    5061 generic.go:356] "Generic (PLEG): container finished" podID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerID="e762852e18318669d25b77e591ed42b08dbead4cce0921e7d4003e845638293c" exitCode=0
Oct 30 03:25:32.659578 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.659553    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" event={"ID":"87f1f00e-6654-4a46-aa16-6e3a38508d6c","Type":"ContainerDied","Data":"e762852e18318669d25b77e591ed42b08dbead4cce0921e7d4003e845638293c"}
Oct 30 03:25:32.659676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.659661    5061 scope.go:122] "RemoveContainer" containerID="b87e8ba7fa42a0a357f1755c211ebd0b0ffc8d20d5ad5a1e6e547d361532a461"
Oct 30 03:25:32.689056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.689033    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 03:25:32.714681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.714655    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-kube-api-access-htvdj\" (UniqueName: \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-kube-api-access-htvdj\") pod \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") "
Oct 30 03:25:32.714804 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.714724    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-token\" (UniqueName: \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-token\") pod \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") "
Oct 30 03:25:32.714804 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.714775    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-socket\") pod \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") "
Oct 30 03:25:32.714937 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.714869    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-credential-socket\") pod \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") "
Oct 30 03:25:32.714937 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.714893    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-podinfo\") pod \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") "
Oct 30 03:25:32.715147 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.714969    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istiod-ca-cert\") pod \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") "
Oct 30 03:25:32.715147 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.715118    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-socket" pod "87f1f00e-6654-4a46-aa16-6e3a38508d6c" (UID: "87f1f00e-6654-4a46-aa16-6e3a38508d6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:32.715276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.715162    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-envoy\") pod \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") "
Oct 30 03:25:32.715276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.715202    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-data\") pod \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") "
Oct 30 03:25:32.715276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.715234    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-credential-socket" pod "87f1f00e-6654-4a46-aa16-6e3a38508d6c" (UID: "87f1f00e-6654-4a46-aa16-6e3a38508d6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:32.715276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.715265    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-certs\") pod \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\" (UID: \"87f1f00e-6654-4a46-aa16-6e3a38508d6c\") "
Oct 30 03:25:32.715551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.715527    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istiod-ca-cert" pod "87f1f00e-6654-4a46-aa16-6e3a38508d6c" (UID: "87f1f00e-6654-4a46-aa16-6e3a38508d6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:32.715681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.715660    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-data" pod "87f1f00e-6654-4a46-aa16-6e3a38508d6c" (UID: "87f1f00e-6654-4a46-aa16-6e3a38508d6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:32.715782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.715764    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-certs" pod "87f1f00e-6654-4a46-aa16-6e3a38508d6c" (UID: "87f1f00e-6654-4a46-aa16-6e3a38508d6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:32.716054 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.716031    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.716134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.716059    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.716134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.716077    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.716134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.716091    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.716134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.716105    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.717414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.717373    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-token" pod "87f1f00e-6654-4a46-aa16-6e3a38508d6c" (UID: "87f1f00e-6654-4a46-aa16-6e3a38508d6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:32.718836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.718802    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-podinfo" pod "87f1f00e-6654-4a46-aa16-6e3a38508d6c" (UID: "87f1f00e-6654-4a46-aa16-6e3a38508d6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:25:32.719215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.719184    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-kube-api-access-htvdj" pod "87f1f00e-6654-4a46-aa16-6e3a38508d6c" (UID: "87f1f00e-6654-4a46-aa16-6e3a38508d6c"). InnerVolumeSpecName "kube-api-access-htvdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:32.723923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.723890    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-envoy" pod "87f1f00e-6654-4a46-aa16-6e3a38508d6c" (UID: "87f1f00e-6654-4a46-aa16-6e3a38508d6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:32.817488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.817452    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htvdj\" (UniqueName: \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-kube-api-access-htvdj\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.817488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.817485    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.817488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.817497    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.817737 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:32.817509    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f1f00e-6654-4a46-aa16-6e3a38508d6c-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:33.298551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:33.298517    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:33.579370 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:33.579285    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:33.667913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:33.667880    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"
Oct 30 03:25:33.667913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:33.667893    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q" event={"ID":"87f1f00e-6654-4a46-aa16-6e3a38508d6c","Type":"ContainerDied","Data":"cb6b8b4a12685103dea32246c5b668f898ef2e0b78a12a667d4634d4fe3af9fc"}
Oct 30 03:25:33.668147 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:33.667950    5061 scope.go:122] "RemoveContainer" containerID="e762852e18318669d25b77e591ed42b08dbead4cce0921e7d4003e845638293c"
Oct 30 03:25:33.749884 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:33.749836    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"]
Oct 30 03:25:33.754243 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:33.754218    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-810119-openshift-default-869dbf496f-fkd8q"]
Oct 30 03:25:34.847776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:34.847741    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" path="/var/lib/kubelet/pods/87f1f00e-6654-4a46-aa16-6e3a38508d6c/volumes"
Oct 30 03:25:35.292748 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:35.292612    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:36.965630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:36.965600    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:38.479738 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:38.479705    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:40.096145 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:40.096108    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:41.781915 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:41.781871    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:43.922077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:43.922039    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:45.813121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:45.813079    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:47.415578 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:47.415535    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:48.987417 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:48.987385    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:50.223168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:50.223120    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:50.788497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:50.788458    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:52.684721 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:52.684683    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:54.246776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:54.246692    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:55.412702 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:55.412620    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:25:55.416380 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:55.416318    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:25:55.418153 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:55.418100    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:25:55.432180 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:55.422487    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:25:55.472381 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:25:55.472275    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:25:55.567162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:55.567065    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"]
Oct 30 03:25:55.567743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:55.567685    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps" podUID="2c362050-6a7f-4339-b028-8ac00f20cc9e" containerName="maas-api" containerID="cri-o://c85d16c4f35915ffebb5edf529b7470448037289e393ae30855e838a9086dd7e" gracePeriod=30
Oct 30 03:25:55.657368 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:55.657269    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:56.111826 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.099374    5061 generic.go:356] "Generic (PLEG): container finished" podID="2c362050-6a7f-4339-b028-8ac00f20cc9e" containerID="c85d16c4f35915ffebb5edf529b7470448037289e393ae30855e838a9086dd7e" exitCode=0
Oct 30 03:25:56.111826 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.099891    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps" event={"ID":"2c362050-6a7f-4339-b028-8ac00f20cc9e","Type":"ContainerDied","Data":"c85d16c4f35915ffebb5edf529b7470448037289e393ae30855e838a9086dd7e"}
Oct 30 03:25:56.179634 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.179602    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 03:25:56.255354 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.255305    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-metrics-certs\") pod \"2c362050-6a7f-4339-b028-8ac00f20cc9e\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") "
Oct 30 03:25:56.255570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.255434    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c362050-6a7f-4339-b028-8ac00f20cc9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c362050-6a7f-4339-b028-8ac00f20cc9e-openshift-service-ca\") pod \"2c362050-6a7f-4339-b028-8ac00f20cc9e\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") "
Oct 30 03:25:56.255570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.255478    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-maas-api-tls\") pod \"2c362050-6a7f-4339-b028-8ac00f20cc9e\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") "
Oct 30 03:25:56.255719 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.255690    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c362050-6a7f-4339-b028-8ac00f20cc9e-kube-api-access-c9rs4\" (UniqueName: \"kubernetes.io/projected/2c362050-6a7f-4339-b028-8ac00f20cc9e-kube-api-access-c9rs4\") pod \"2c362050-6a7f-4339-b028-8ac00f20cc9e\" (UID: \"2c362050-6a7f-4339-b028-8ac00f20cc9e\") "
Oct 30 03:25:56.255908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.255876    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c362050-6a7f-4339-b028-8ac00f20cc9e-openshift-service-ca" pod "2c362050-6a7f-4339-b028-8ac00f20cc9e" (UID: "2c362050-6a7f-4339-b028-8ac00f20cc9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:56.256240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.256220    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c362050-6a7f-4339-b028-8ac00f20cc9e-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:56.258534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.258338    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-metrics-certs" pod "2c362050-6a7f-4339-b028-8ac00f20cc9e" (UID: "2c362050-6a7f-4339-b028-8ac00f20cc9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:56.258534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.258419    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c362050-6a7f-4339-b028-8ac00f20cc9e-kube-api-access-c9rs4" pod "2c362050-6a7f-4339-b028-8ac00f20cc9e" (UID: "2c362050-6a7f-4339-b028-8ac00f20cc9e"). InnerVolumeSpecName "kube-api-access-c9rs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:56.258534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.258469    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-maas-api-tls" pod "2c362050-6a7f-4339-b028-8ac00f20cc9e" (UID: "2c362050-6a7f-4339-b028-8ac00f20cc9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:56.346253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.346218    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:56.357269 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.357233    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:56.357269 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.357271    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9rs4\" (UniqueName: \"kubernetes.io/projected/2c362050-6a7f-4339-b028-8ac00f20cc9e-kube-api-access-c9rs4\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:56.357416 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.357282    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c362050-6a7f-4339-b028-8ac00f20cc9e-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:56.482284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.482187    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:56.623780 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.623738    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:56.757697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:56.757604    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:57.069122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:57.069039    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:57.120020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:57.119925    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps" event={"ID":"2c362050-6a7f-4339-b028-8ac00f20cc9e","Type":"ContainerDied","Data":"89d727590e9cb47fad106a8a33ab83a1bea21a4b89d023264e5f6ff03fff3082"}
Oct 30 03:25:57.120020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:57.120029    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"
Oct 30 03:25:57.120559 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:57.120111    5061 scope.go:122] "RemoveContainer" containerID="c85d16c4f35915ffebb5edf529b7470448037289e393ae30855e838a9086dd7e"
Oct 30 03:25:57.129661 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:57.129601    5061 scope.go:122] "RemoveContainer" containerID="c85d16c4f35915ffebb5edf529b7470448037289e393ae30855e838a9086dd7e"
Oct 30 03:25:57.141400 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:25:57.141339    5061 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_maas-api_maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps_odh-ai-gateway-infra_2c362050-6a7f-4339-b028-8ac00f20cc9e_0 in pod sandbox 89d727590e9cb47fad106a8a33ab83a1bea21a4b89d023264e5f6ff03fff3082 from index: no such id: 'c85d16c4f35915ffebb5edf529b7470448037289e393ae30855e838a9086dd7e'" containerID="c85d16c4f35915ffebb5edf529b7470448037289e393ae30855e838a9086dd7e"
Oct 30 03:25:57.141714 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:57.141430    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85d16c4f35915ffebb5edf529b7470448037289e393ae30855e838a9086dd7e"} err="rpc error: code = Unknown desc = failed to delete container k8s_maas-api_maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps_odh-ai-gateway-infra_2c362050-6a7f-4339-b028-8ac00f20cc9e_0 in pod sandbox 89d727590e9cb47fad106a8a33ab83a1bea21a4b89d023264e5f6ff03fff3082 from index: no such id: 'c85d16c4f35915ffebb5edf529b7470448037289e393ae30855e838a9086dd7e'"
Oct 30 03:25:57.177517 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:57.177463    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"]
Oct 30 03:25:57.184096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:57.184050    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7d1788-5f95676695-2rlps"]
Oct 30 03:25:58.375652 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.375619    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:58.380219 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.380193    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"]
Oct 30 03:25:58.380891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.380869    5061 cpu_manager.go:414] "containerMap: removing container" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy"
Oct 30 03:25:58.381000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.380892    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy"
Oct 30 03:25:58.381000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.380903    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8bf611f0-2273-4646-8cfc-bd12ec8a2dee" containerName="main"
Oct 30 03:25:58.381000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.380911    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf611f0-2273-4646-8cfc-bd12ec8a2dee" containerName="main"
Oct 30 03:25:58.381000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.380962    5061 cpu_manager.go:414] "containerMap: removing container" podUID="c24ea1dc-460b-4d52-83d5-17c97ca3e29b" containerName="payload-processing"
Oct 30 03:25:58.381000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.380970    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24ea1dc-460b-4d52-83d5-17c97ca3e29b" containerName="payload-processing"
Oct 30 03:25:58.381268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381004    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2c362050-6a7f-4339-b028-8ac00f20cc9e" containerName="maas-api"
Oct 30 03:25:58.381268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381012    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c362050-6a7f-4339-b028-8ac00f20cc9e" containerName="maas-api"
Oct 30 03:25:58.381268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381026    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main"
Oct 30 03:25:58.381268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381033    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main"
Oct 30 03:25:58.381268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381058    5061 cpu_manager.go:414] "containerMap: removing container" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy"
Oct 30 03:25:58.381268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381066    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy"
Oct 30 03:25:58.381268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381083    5061 cpu_manager.go:414] "containerMap: removing container" podUID="c12a8d12-5c1d-492c-a468-a65a8f06fadf" containerName="payload-pre-processing"
Oct 30 03:25:58.381268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381091    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12a8d12-5c1d-492c-a468-a65a8f06fadf" containerName="payload-pre-processing"
Oct 30 03:25:58.381268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381248    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy"
Oct 30 03:25:58.381268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381259    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf611f0-2273-4646-8cfc-bd12ec8a2dee" containerName="main"
Oct 30 03:25:58.381752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381275    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24ea1dc-460b-4d52-83d5-17c97ca3e29b" containerName="payload-processing"
Oct 30 03:25:58.381752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381289    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c362050-6a7f-4339-b028-8ac00f20cc9e" containerName="maas-api"
Oct 30 03:25:58.381752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381297    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1318fb59-a096-440a-b3cb-43139a3bc387" containerName="main"
Oct 30 03:25:58.381752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381308    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f1f00e-6654-4a46-aa16-6e3a38508d6c" containerName="istio-proxy"
Oct 30 03:25:58.381752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.381316    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12a8d12-5c1d-492c-a468-a65a8f06fadf" containerName="payload-pre-processing"
Oct 30 03:25:58.386613 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.386593    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.404308 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.404282    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"]
Oct 30 03:25:58.411130 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.411106    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-4b5459\"/\"kube-root-ca.crt\""
Oct 30 03:25:58.411130 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.411107    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-4b5459\"/\"openshift-service-ca.crt\""
Oct 30 03:25:58.411265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.411114    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-4b5459\"/\"default-dockercfg-j7g97\""
Oct 30 03:25:58.413640 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.413619    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-4b5459\"/\"rate-test-model-4b5459-kserve-self-signed-certs\""
Oct 30 03:25:58.439348 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.439325    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"]
Oct 30 03:25:58.439505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.439487    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4" podUID="83428494-94f2-427a-a101-2cc2d4f45a24" containerName="payload-processing" containerID="cri-o://e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56" gracePeriod=30
Oct 30 03:25:58.458753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.458727    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"]
Oct 30 03:25:58.458964 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.458945    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952" podUID="aae61a65-0a49-4b8b-935f-45bb6981e8ba" containerName="payload-pre-processing" containerID="cri-o://2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5" gracePeriod=30
Oct 30 03:25:58.488553 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.488521    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tls-certs\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.488693 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.488576    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tmp-dir\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.488693 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.488681    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-dshm\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.488788 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.488751    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-home\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.488925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.488901    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-model-cache\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.489009 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.488979    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdp6c\" (UniqueName: \"kubernetes.io/projected/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-kube-api-access-jdp6c\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.508811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.508780    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:58.590186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.590153    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-home\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.590319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.590210    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-model-cache\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.590319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.590252    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdp6c\" (UniqueName: \"kubernetes.io/projected/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-kube-api-access-jdp6c\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.590414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.590394    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tls-certs\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.590456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.590444    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tmp-dir\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.590511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.590489    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-dshm\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.590577 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.590558    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-home\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.590631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.590603    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-model-cache\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.590831 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.590812    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tmp-dir\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.593171 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.593146    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-dshm\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.594128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.594095    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tls-certs\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.609381 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.609356    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdp6c\" (UniqueName: \"kubernetes.io/projected/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-kube-api-access-jdp6c\") pod \"rate-test-model-4b5459-kserve-54c4c77989-9vk6d\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") " pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.635345 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.633961    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:58.700178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.700153    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:25:58.721076 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.714810    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"]
Oct 30 03:25:58.721076 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.715049    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h" podUID="0166ec2f-c5c8-4387-91a7-15093053f4c4" containerName="maas-api" containerID="cri-o://a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983" gracePeriod=30
Oct 30 03:25:58.796205 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.796173    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:58.835732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.835703    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:25:58.841916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.839115    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:25:58.850019 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.849993    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c362050-6a7f-4339-b028-8ac00f20cc9e" path="/var/lib/kubelet/pods/2c362050-6a7f-4339-b028-8ac00f20cc9e/volumes"
Oct 30 03:25:58.888384 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.888307    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"]
Oct 30 03:25:58.888566 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.888536    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5" podUID="b997ff8f-09d5-44e8-8c66-3c89ff54b04f" containerName="payload-processing" containerID="cri-o://0de491ddf8444d1048051b74fbf7815980b09bc787a18e9a7d6069f2fb4347cc" gracePeriod=30
Oct 30 03:25:58.895004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.894974    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae61a65-0a49-4b8b-935f-45bb6981e8ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae61a65-0a49-4b8b-935f-45bb6981e8ba-plugins-config-volume\") pod \"aae61a65-0a49-4b8b-935f-45bb6981e8ba\" (UID: \"aae61a65-0a49-4b8b-935f-45bb6981e8ba\") "
Oct 30 03:25:58.895121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.895057    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83428494-94f2-427a-a101-2cc2d4f45a24-kube-api-access-zrzvc\" (UniqueName: \"kubernetes.io/projected/83428494-94f2-427a-a101-2cc2d4f45a24-kube-api-access-zrzvc\") pod \"83428494-94f2-427a-a101-2cc2d4f45a24\" (UID: \"83428494-94f2-427a-a101-2cc2d4f45a24\") "
Oct 30 03:25:58.895121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.895103    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae61a65-0a49-4b8b-935f-45bb6981e8ba-kube-api-access-vtw7s\" (UniqueName: \"kubernetes.io/projected/aae61a65-0a49-4b8b-935f-45bb6981e8ba-kube-api-access-vtw7s\") pod \"aae61a65-0a49-4b8b-935f-45bb6981e8ba\" (UID: \"aae61a65-0a49-4b8b-935f-45bb6981e8ba\") "
Oct 30 03:25:58.895195 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.895153    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83428494-94f2-427a-a101-2cc2d4f45a24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83428494-94f2-427a-a101-2cc2d4f45a24-plugins-config-volume\") pod \"83428494-94f2-427a-a101-2cc2d4f45a24\" (UID: \"83428494-94f2-427a-a101-2cc2d4f45a24\") "
Oct 30 03:25:58.896038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.896018    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae61a65-0a49-4b8b-935f-45bb6981e8ba-plugins-config-volume" pod "aae61a65-0a49-4b8b-935f-45bb6981e8ba" (UID: "aae61a65-0a49-4b8b-935f-45bb6981e8ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:58.897994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.897971    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83428494-94f2-427a-a101-2cc2d4f45a24-kube-api-access-zrzvc" pod "83428494-94f2-427a-a101-2cc2d4f45a24" (UID: "83428494-94f2-427a-a101-2cc2d4f45a24"). InnerVolumeSpecName "kube-api-access-zrzvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:58.899366 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.899234    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83428494-94f2-427a-a101-2cc2d4f45a24-plugins-config-volume" pod "83428494-94f2-427a-a101-2cc2d4f45a24" (UID: "83428494-94f2-427a-a101-2cc2d4f45a24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:58.900270 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.900244    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae61a65-0a49-4b8b-935f-45bb6981e8ba-kube-api-access-vtw7s" pod "aae61a65-0a49-4b8b-935f-45bb6981e8ba" (UID: "aae61a65-0a49-4b8b-935f-45bb6981e8ba"). InnerVolumeSpecName "kube-api-access-vtw7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:58.932230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.932202    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:58.997482 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.997448    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae61a65-0a49-4b8b-935f-45bb6981e8ba-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:58.997482 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.997483    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrzvc\" (UniqueName: \"kubernetes.io/projected/83428494-94f2-427a-a101-2cc2d4f45a24-kube-api-access-zrzvc\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:58.997673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.997500    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtw7s\" (UniqueName: \"kubernetes.io/projected/aae61a65-0a49-4b8b-935f-45bb6981e8ba-kube-api-access-vtw7s\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:58.997673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:58.997515    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83428494-94f2-427a-a101-2cc2d4f45a24-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:59.029964 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.029933    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:25:59.065051 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.064980    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"]
Oct 30 03:25:59.075120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.075044    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:59.099843 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.098683    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0166ec2f-c5c8-4387-91a7-15093053f4c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0166ec2f-c5c8-4387-91a7-15093053f4c4-openshift-service-ca\") pod \"0166ec2f-c5c8-4387-91a7-15093053f4c4\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") "
Oct 30 03:25:59.099843 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.098803    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0166ec2f-c5c8-4387-91a7-15093053f4c4-kube-api-access-vk55m\" (UniqueName: \"kubernetes.io/projected/0166ec2f-c5c8-4387-91a7-15093053f4c4-kube-api-access-vk55m\") pod \"0166ec2f-c5c8-4387-91a7-15093053f4c4\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") "
Oct 30 03:25:59.099843 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.098949    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-metrics-certs\") pod \"0166ec2f-c5c8-4387-91a7-15093053f4c4\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") "
Oct 30 03:25:59.099843 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.099153    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-maas-api-tls\") pod \"0166ec2f-c5c8-4387-91a7-15093053f4c4\" (UID: \"0166ec2f-c5c8-4387-91a7-15093053f4c4\") "
Oct 30 03:25:59.100286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.099902    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0166ec2f-c5c8-4387-91a7-15093053f4c4-openshift-service-ca" pod "0166ec2f-c5c8-4387-91a7-15093053f4c4" (UID: "0166ec2f-c5c8-4387-91a7-15093053f4c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:59.102140 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.102117    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0166ec2f-c5c8-4387-91a7-15093053f4c4-kube-api-access-vk55m" pod "0166ec2f-c5c8-4387-91a7-15093053f4c4" (UID: "0166ec2f-c5c8-4387-91a7-15093053f4c4"). InnerVolumeSpecName "kube-api-access-vk55m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:59.102140 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.102128    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-metrics-certs" pod "0166ec2f-c5c8-4387-91a7-15093053f4c4" (UID: "0166ec2f-c5c8-4387-91a7-15093053f4c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:59.103161 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.103135    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-maas-api-tls" pod "0166ec2f-c5c8-4387-91a7-15093053f4c4" (UID: "0166ec2f-c5c8-4387-91a7-15093053f4c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:59.139603 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.139535    5061 generic.go:356] "Generic (PLEG): container finished" podID="b997ff8f-09d5-44e8-8c66-3c89ff54b04f" containerID="0de491ddf8444d1048051b74fbf7815980b09bc787a18e9a7d6069f2fb4347cc" exitCode=0
Oct 30 03:25:59.139724 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.139617    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5" event={"ID":"b997ff8f-09d5-44e8-8c66-3c89ff54b04f","Type":"ContainerDied","Data":"0de491ddf8444d1048051b74fbf7815980b09bc787a18e9a7d6069f2fb4347cc"}
Oct 30 03:25:59.141317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.141294    5061 generic.go:356] "Generic (PLEG): container finished" podID="aae61a65-0a49-4b8b-935f-45bb6981e8ba" containerID="2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5" exitCode=0
Oct 30 03:25:59.142914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.141331    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"
Oct 30 03:25:59.142914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.141351    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952" event={"ID":"aae61a65-0a49-4b8b-935f-45bb6981e8ba","Type":"ContainerDied","Data":"2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5"}
Oct 30 03:25:59.142914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.141677    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952" event={"ID":"aae61a65-0a49-4b8b-935f-45bb6981e8ba","Type":"ContainerDied","Data":"f7b97ed5c2e63376343b138179d1088383f2ffd9d3bfda2079ee5db6a8a7eda6"}
Oct 30 03:25:59.142914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.141713    5061 scope.go:122] "RemoveContainer" containerID="2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5"
Oct 30 03:25:59.145581 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.145554    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d" event={"ID":"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7","Type":"ContainerStarted","Data":"e18fd30e0deefe9b5cb7896f606bbdfc47831103da9bdcb7b5f3e6731627168b"}
Oct 30 03:25:59.150774 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.146989    5061 generic.go:356] "Generic (PLEG): container finished" podID="83428494-94f2-427a-a101-2cc2d4f45a24" containerID="e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56" exitCode=0
Oct 30 03:25:59.150774 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.147059    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4" event={"ID":"83428494-94f2-427a-a101-2cc2d4f45a24","Type":"ContainerDied","Data":"e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56"}
Oct 30 03:25:59.150774 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.147098    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4" event={"ID":"83428494-94f2-427a-a101-2cc2d4f45a24","Type":"ContainerDied","Data":"c268e2076710f143d616a9c52ca32e955f3d0020b41ef9a83771d6079e50c921"}
Oct 30 03:25:59.150774 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.147171    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"
Oct 30 03:25:59.152402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.152377    5061 generic.go:356] "Generic (PLEG): container finished" podID="0166ec2f-c5c8-4387-91a7-15093053f4c4" containerID="a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983" exitCode=0
Oct 30 03:25:59.152502 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.152455    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h" event={"ID":"0166ec2f-c5c8-4387-91a7-15093053f4c4","Type":"ContainerDied","Data":"a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983"}
Oct 30 03:25:59.152549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.152499    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h" event={"ID":"0166ec2f-c5c8-4387-91a7-15093053f4c4","Type":"ContainerDied","Data":"4bfd6f8239f64e862983900efd5a7429ab8e8576efcd8f6c1dc35241921dbcb4"}
Oct 30 03:25:59.152607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.152590    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"
Oct 30 03:25:59.158699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.158681    5061 scope.go:122] "RemoveContainer" containerID="2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5"
Oct 30 03:25:59.161830 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:25:59.161679    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5\": container with ID starting with 2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5 not found: ID does not exist" containerID="2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5"
Oct 30 03:25:59.161830 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.161714    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5"} err="failed to get container status \"2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5\": rpc error: code = NotFound desc = could not find container \"2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5\": container with ID starting with 2d38b9eaccbef7e3cc2e1aa56238713f3ecfafcea103d2485f24c8209ea5e9a5 not found: ID does not exist"
Oct 30 03:25:59.161830 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.161738    5061 scope.go:122] "RemoveContainer" containerID="e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56"
Oct 30 03:25:59.174529 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.174509    5061 scope.go:122] "RemoveContainer" containerID="e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56"
Oct 30 03:25:59.174836 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:25:59.174799    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56\": container with ID starting with e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56 not found: ID does not exist" containerID="e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56"
Oct 30 03:25:59.174920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.174840    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56"} err="failed to get container status \"e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56\": rpc error: code = NotFound desc = could not find container \"e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56\": container with ID starting with e4d76d9d6f469935a4eec289d90ee50a1a729991d1590566c430c5390b726c56 not found: ID does not exist"
Oct 30 03:25:59.174920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.174873    5061 scope.go:122] "RemoveContainer" containerID="a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983"
Oct 30 03:25:59.177870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.177837    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:25:59.184601 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.184582    5061 scope.go:122] "RemoveContainer" containerID="a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983"
Oct 30 03:25:59.184984 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:25:59.184910    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983\": container with ID starting with a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983 not found: ID does not exist" containerID="a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983"
Oct 30 03:25:59.185107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.185086    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983"} err="failed to get container status \"a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983\": rpc error: code = NotFound desc = could not find container \"a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983\": container with ID starting with a3763111f19bef57b5a1e436ccd9924b30eb03076460af8b389e09201006d983 not found: ID does not exist"
Oct 30 03:25:59.201727 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.201693    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk55m\" (UniqueName: \"kubernetes.io/projected/0166ec2f-c5c8-4387-91a7-15093053f4c4-kube-api-access-vk55m\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:59.228882 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.205064    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:59.228882 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.205096    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0166ec2f-c5c8-4387-91a7-15093053f4c4-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:59.228882 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.205110    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0166ec2f-c5c8-4387-91a7-15093053f4c4-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:59.280994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.280960    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"]
Oct 30 03:25:59.306592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.306513    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-plugins-config-volume\") pod \"b997ff8f-09d5-44e8-8c66-3c89ff54b04f\" (UID: \"b997ff8f-09d5-44e8-8c66-3c89ff54b04f\") "
Oct 30 03:25:59.306812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.306796    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-kube-api-access-sl6t9\" (UniqueName: \"kubernetes.io/projected/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-kube-api-access-sl6t9\") pod \"b997ff8f-09d5-44e8-8c66-3c89ff54b04f\" (UID: \"b997ff8f-09d5-44e8-8c66-3c89ff54b04f\") "
Oct 30 03:25:59.309312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.309283    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-plugins-config-volume" pod "b997ff8f-09d5-44e8-8c66-3c89ff54b04f" (UID: "b997ff8f-09d5-44e8-8c66-3c89ff54b04f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:59.315809 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.315783    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-kube-api-access-sl6t9" pod "b997ff8f-09d5-44e8-8c66-3c89ff54b04f" (UID: "b997ff8f-09d5-44e8-8c66-3c89ff54b04f"). InnerVolumeSpecName "kube-api-access-sl6t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:59.320329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.320304    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a0f7e-6db469bc78-x7l6h"]
Oct 30 03:25:59.327535 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.327508    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"]
Oct 30 03:25:59.330609 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.330589    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-146bb7-746dc644cf-d5952"]
Oct 30 03:25:59.349685 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.349666    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"]
Oct 30 03:25:59.355230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.355205    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-146bb7-798dcd9494-62zq4"]
Oct 30 03:25:59.408406 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.408341    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:59.408406 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.408367    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl6t9\" (UniqueName: \"kubernetes.io/projected/b997ff8f-09d5-44e8-8c66-3c89ff54b04f-kube-api-access-sl6t9\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:25:59.470066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.470040    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:59.551148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.551119    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"]
Oct 30 03:25:59.551328 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.551307    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9" podUID="1bce83b4-dca3-4ad5-a05e-940006d42ac3" containerName="istio-proxy" containerID="cri-o://875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42" gracePeriod=30
Oct 30 03:25:59.597843 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.597813    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:59.726738 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.717032    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:59.847134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:25:59.847076    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:00.160240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.160204    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d" event={"ID":"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7","Type":"ContainerStarted","Data":"befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd"}
Oct 30 03:26:00.163261 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.163240    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"
Oct 30 03:26:00.163425 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.163249    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5" event={"ID":"b997ff8f-09d5-44e8-8c66-3c89ff54b04f","Type":"ContainerDied","Data":"18834286eeee1587ceff567843451ac9d3303fde4c3db3ad96a778a3891f6732"}
Oct 30 03:26:00.163425 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.163317    5061 scope.go:122] "RemoveContainer" containerID="0de491ddf8444d1048051b74fbf7815980b09bc787a18e9a7d6069f2fb4347cc"
Oct 30 03:26:00.234953 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.234903    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d" podStartSLOduration=2.234889745 podStartE2EDuration="2.234889745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:00.232463392 +0000 UTC m=+2705.534641757" watchObservedRunningTime="2026-10-30 03:26:00.234889745 +0000 UTC m=+2705.537068111"
Oct 30 03:26:00.266630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.266598    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"]
Oct 30 03:26:00.271904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.271880    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a0f7e-55577bf768-j2nh5"]
Oct 30 03:26:00.281890 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.281832    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:00.701280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.701250    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:26:00.849444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.849410    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0166ec2f-c5c8-4387-91a7-15093053f4c4" path="/var/lib/kubelet/pods/0166ec2f-c5c8-4387-91a7-15093053f4c4/volumes"
Oct 30 03:26:00.849840 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.849826    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83428494-94f2-427a-a101-2cc2d4f45a24" path="/var/lib/kubelet/pods/83428494-94f2-427a-a101-2cc2d4f45a24/volumes"
Oct 30 03:26:00.850217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.850204    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae61a65-0a49-4b8b-935f-45bb6981e8ba" path="/var/lib/kubelet/pods/aae61a65-0a49-4b8b-935f-45bb6981e8ba/volumes"
Oct 30 03:26:00.850879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:00.850861    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b997ff8f-09d5-44e8-8c66-3c89ff54b04f" path="/var/lib/kubelet/pods/b997ff8f-09d5-44e8-8c66-3c89ff54b04f/volumes"
Oct 30 03:26:01.518924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:01.518882    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:03.255778 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:03.255693    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:03.382168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:03.382135    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:03.519880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:03.519780    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:03.763723 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:03.763693    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"]
Oct 30 03:26:03.763942 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:03.763908    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy" containerID="cri-o://ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6" gracePeriod=30
Oct 30 03:26:03.873888 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:03.873824    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:04.822520 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.822462    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:26:04.885163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.885121    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-token\" (UniqueName: \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-token\") pod \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") "
Oct 30 03:26:04.885393 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.885375    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-podinfo\") pod \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") "
Oct 30 03:26:04.885549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.885531    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-data\") pod \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") "
Oct 30 03:26:04.885805 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.885781    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-envoy\") pod \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") "
Oct 30 03:26:04.885920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.885860    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istiod-ca-cert\") pod \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") "
Oct 30 03:26:04.886046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.885973    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-credential-socket\") pod \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") "
Oct 30 03:26:04.886046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.885998    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-data" pod "1bce83b4-dca3-4ad5-a05e-940006d42ac3" (UID: "1bce83b4-dca3-4ad5-a05e-940006d42ac3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:04.886168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.886129    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-socket\") pod \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") "
Oct 30 03:26:04.886522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.886248    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-certs\") pod \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") "
Oct 30 03:26:04.886522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.886473    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-socket" pod "1bce83b4-dca3-4ad5-a05e-940006d42ac3" (UID: "1bce83b4-dca3-4ad5-a05e-940006d42ac3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:04.886522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.886491    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istiod-ca-cert" pod "1bce83b4-dca3-4ad5-a05e-940006d42ac3" (UID: "1bce83b4-dca3-4ad5-a05e-940006d42ac3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:04.886760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.886562    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-credential-socket" pod "1bce83b4-dca3-4ad5-a05e-940006d42ac3" (UID: "1bce83b4-dca3-4ad5-a05e-940006d42ac3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:04.886760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.886620    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-kube-api-access-xgjft\" (UniqueName: \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-kube-api-access-xgjft\") pod \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\" (UID: \"1bce83b4-dca3-4ad5-a05e-940006d42ac3\") "
Oct 30 03:26:04.887484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.887456    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-certs" pod "1bce83b4-dca3-4ad5-a05e-940006d42ac3" (UID: "1bce83b4-dca3-4ad5-a05e-940006d42ac3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:04.888302 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.888276    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:04.888407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.888308    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:04.888407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.888315    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-podinfo" pod "1bce83b4-dca3-4ad5-a05e-940006d42ac3" (UID: "1bce83b4-dca3-4ad5-a05e-940006d42ac3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:26:04.888407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.888324    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:04.888407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.888339    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:04.888407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.888358    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:04.890905 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.889417    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-token" pod "1bce83b4-dca3-4ad5-a05e-940006d42ac3" (UID: "1bce83b4-dca3-4ad5-a05e-940006d42ac3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:04.901075 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.901049    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-envoy" pod "1bce83b4-dca3-4ad5-a05e-940006d42ac3" (UID: "1bce83b4-dca3-4ad5-a05e-940006d42ac3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:04.903300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.903273    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-kube-api-access-xgjft" pod "1bce83b4-dca3-4ad5-a05e-940006d42ac3" (UID: "1bce83b4-dca3-4ad5-a05e-940006d42ac3"). InnerVolumeSpecName "kube-api-access-xgjft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:04.989765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.989733    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgjft\" (UniqueName: \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-kube-api-access-xgjft\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:04.989765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.989761    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:04.989765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.989771    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:04.990034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:04.989780    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bce83b4-dca3-4ad5-a05e-940006d42ac3-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:05.201935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.201831    5061 generic.go:356] "Generic (PLEG): container finished" podID="1bce83b4-dca3-4ad5-a05e-940006d42ac3" containerID="875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42" exitCode=0
Oct 30 03:26:05.202100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.201939    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"
Oct 30 03:26:05.202100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.201963    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9" event={"ID":"1bce83b4-dca3-4ad5-a05e-940006d42ac3","Type":"ContainerDied","Data":"875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42"}
Oct 30 03:26:05.202100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.202051    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9" event={"ID":"1bce83b4-dca3-4ad5-a05e-940006d42ac3","Type":"ContainerDied","Data":"526e7742811201115ada288e974ab32cdc0bc110825102ff928647c244070cab"}
Oct 30 03:26:05.202100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.202078    5061 scope.go:122] "RemoveContainer" containerID="875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42"
Oct 30 03:26:05.213923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.213905    5061 scope.go:122] "RemoveContainer" containerID="875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42"
Oct 30 03:26:05.214222 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:26:05.214191    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42\": container with ID starting with 875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42 not found: ID does not exist" containerID="875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42"
Oct 30 03:26:05.214348 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.214230    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42"} err="failed to get container status \"875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42\": rpc error: code = NotFound desc = could not find container \"875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42\": container with ID starting with 875bdc7269351d99510ddf994e8ee2aca77fd83702a4026f08f9acff57890a42 not found: ID does not exist"
Oct 30 03:26:05.247280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.247256    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"]
Oct 30 03:26:05.249995 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.249975    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d3552-openshift-default-758cc86dd7-chtm9"]
Oct 30 03:26:05.297484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.297453    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:05.753420 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.753369    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:05.875112 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:05.875079    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:06.068421 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:06.068393    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:06.849518 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:06.849484    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bce83b4-dca3-4ad5-a05e-940006d42ac3" path="/var/lib/kubelet/pods/1bce83b4-dca3-4ad5-a05e-940006d42ac3/volumes"
Oct 30 03:26:07.212330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:07.212205    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:07.367201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:07.367149    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:07.510091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:07.509986    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:08.499165 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:08.499125    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:08.635756 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:08.635723    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:08.700572 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:08.700540    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:26:08.713347 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:08.713324    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:26:09.069274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.069246    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 03:26:09.193903 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.193839    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istiod-ca-cert\") pod \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") "
Oct 30 03:26:09.194103 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.193933    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-kube-api-access-4dcsf\" (UniqueName: \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-kube-api-access-4dcsf\") pod \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") "
Oct 30 03:26:09.194103 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.193992    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-socket\") pod \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") "
Oct 30 03:26:09.194103 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.194020    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-envoy\") pod \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") "
Oct 30 03:26:09.194103 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.194083    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-token\" (UniqueName: \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-token\") pod \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") "
Oct 30 03:26:09.194356 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.194192    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-certs\") pod \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") "
Oct 30 03:26:09.194356 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.194277    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-podinfo\") pod \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") "
Oct 30 03:26:09.194356 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.194306    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-credential-socket\") pod \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") "
Oct 30 03:26:09.194356 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.194334    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-data\") pod \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\" (UID: \"f8eec092-64ae-43a2-8c3a-7c1e0f27b168\") "
Oct 30 03:26:09.195977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.194564    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istiod-ca-cert" pod "f8eec092-64ae-43a2-8c3a-7c1e0f27b168" (UID: "f8eec092-64ae-43a2-8c3a-7c1e0f27b168"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:09.195977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.195166    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.206664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.206627    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-certs" pod "f8eec092-64ae-43a2-8c3a-7c1e0f27b168" (UID: "f8eec092-64ae-43a2-8c3a-7c1e0f27b168"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:09.206887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.206862    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-socket" pod "f8eec092-64ae-43a2-8c3a-7c1e0f27b168" (UID: "f8eec092-64ae-43a2-8c3a-7c1e0f27b168"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:09.207825 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.207802    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-credential-socket" pod "f8eec092-64ae-43a2-8c3a-7c1e0f27b168" (UID: "f8eec092-64ae-43a2-8c3a-7c1e0f27b168"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:09.208015 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.207926    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-data" pod "f8eec092-64ae-43a2-8c3a-7c1e0f27b168" (UID: "f8eec092-64ae-43a2-8c3a-7c1e0f27b168"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:09.209587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.209560    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-kube-api-access-4dcsf" pod "f8eec092-64ae-43a2-8c3a-7c1e0f27b168" (UID: "f8eec092-64ae-43a2-8c3a-7c1e0f27b168"). InnerVolumeSpecName "kube-api-access-4dcsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:09.215736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.215707    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-token" pod "f8eec092-64ae-43a2-8c3a-7c1e0f27b168" (UID: "f8eec092-64ae-43a2-8c3a-7c1e0f27b168"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:09.216121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.216100    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-podinfo" pod "f8eec092-64ae-43a2-8c3a-7c1e0f27b168" (UID: "f8eec092-64ae-43a2-8c3a-7c1e0f27b168"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:26:09.217008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.216984    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-envoy" pod "f8eec092-64ae-43a2-8c3a-7c1e0f27b168" (UID: "f8eec092-64ae-43a2-8c3a-7c1e0f27b168"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:09.234568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.234546    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"
Oct 30 03:26:09.234676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.234648    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" event={"ID":"f8eec092-64ae-43a2-8c3a-7c1e0f27b168","Type":"ContainerDied","Data":"ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6"}
Oct 30 03:26:09.234756 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.234745    5061 scope.go:122] "RemoveContainer" containerID="ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6"
Oct 30 03:26:09.236929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.236896    5061 generic.go:356] "Generic (PLEG): container finished" podID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerID="ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6" exitCode=0
Oct 30 03:26:09.237294 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.237273    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb" event={"ID":"f8eec092-64ae-43a2-8c3a-7c1e0f27b168","Type":"ContainerDied","Data":"4d457c3f2c07bc4b2dea3be510613e73bdb2f74087165f62d62f7ce884da2bf4"}
Oct 30 03:26:09.245615 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.245587    5061 scope.go:122] "RemoveContainer" containerID="ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6"
Oct 30 03:26:09.245891 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:26:09.245834    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6\": container with ID starting with ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6 not found: ID does not exist" containerID="ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6"
Oct 30 03:26:09.245891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.245878    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6"} err="failed to get container status \"ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6\": rpc error: code = NotFound desc = could not find container \"ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6\": container with ID starting with ddce4434bb96690f969c31a08b6d66c045d912416b194671fbbc8df1f91902f6 not found: ID does not exist"
Oct 30 03:26:09.251141 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.251123    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:26:09.283272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.283252    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"]
Oct 30 03:26:09.288960 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.288941    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7d1788-openshift-default-86d57f97ff-sj9pb"]
Oct 30 03:26:09.296667 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.296619    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dcsf\" (UniqueName: \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-kube-api-access-4dcsf\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.296667 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.296640    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.296667 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.296651    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.296667 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.296659    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.296667 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.296668    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.296933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.296676    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.296933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.296686    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.296933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.296699    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eec092-64ae-43a2-8c3a-7c1e0f27b168-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:09.602403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:09.602327    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:10.849195 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:10.849165    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" path="/var/lib/kubelet/pods/f8eec092-64ae-43a2-8c3a-7c1e0f27b168/volumes"
Oct 30 03:26:11.270002 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:11.269921    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:12.743390 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:12.743358    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:14.299994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:14.299947    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:16.101972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:16.101942    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:18.226420 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:18.226383    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:18.399619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:18.399588    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"]
Oct 30 03:26:18.573023 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:18.572987    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:18.697152 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:18.697116    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:19.858215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:19.858180    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:21.419795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:21.419740    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:21.695788 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:21.695705    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:22.988512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:22.988479    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:23.400917 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:23.400878    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h" podUID="1d00176e-84de-4dd2-bc2e-42d663325e2a" containerName="payload-processing" containerID="cri-o://d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3" gracePeriod=25
Oct 30 03:26:23.691801 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:23.691776    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:26:23.770752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:23.770712    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d00176e-84de-4dd2-bc2e-42d663325e2a-kube-api-access-4t7nk\" (UniqueName: \"kubernetes.io/projected/1d00176e-84de-4dd2-bc2e-42d663325e2a-kube-api-access-4t7nk\") pod \"1d00176e-84de-4dd2-bc2e-42d663325e2a\" (UID: \"1d00176e-84de-4dd2-bc2e-42d663325e2a\") "
Oct 30 03:26:23.771077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:23.771049    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d00176e-84de-4dd2-bc2e-42d663325e2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d00176e-84de-4dd2-bc2e-42d663325e2a-plugins-config-volume\") pod \"1d00176e-84de-4dd2-bc2e-42d663325e2a\" (UID: \"1d00176e-84de-4dd2-bc2e-42d663325e2a\") "
Oct 30 03:26:23.771516 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:23.771487    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d00176e-84de-4dd2-bc2e-42d663325e2a-plugins-config-volume" pod "1d00176e-84de-4dd2-bc2e-42d663325e2a" (UID: "1d00176e-84de-4dd2-bc2e-42d663325e2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:23.771780 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:23.771757    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d00176e-84de-4dd2-bc2e-42d663325e2a-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:23.773114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:23.773087    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d00176e-84de-4dd2-bc2e-42d663325e2a-kube-api-access-4t7nk" pod "1d00176e-84de-4dd2-bc2e-42d663325e2a" (UID: "1d00176e-84de-4dd2-bc2e-42d663325e2a"). InnerVolumeSpecName "kube-api-access-4t7nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:23.873068 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:23.873035    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t7nk\" (UniqueName: \"kubernetes.io/projected/1d00176e-84de-4dd2-bc2e-42d663325e2a-kube-api-access-4t7nk\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:24.340693 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:24.340660    5061 generic.go:356] "Generic (PLEG): container finished" podID="1d00176e-84de-4dd2-bc2e-42d663325e2a" containerID="d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3" exitCode=0
Oct 30 03:26:24.341223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:24.340721    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"
Oct 30 03:26:24.341223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:24.340741    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h" event={"ID":"1d00176e-84de-4dd2-bc2e-42d663325e2a","Type":"ContainerDied","Data":"d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3"}
Oct 30 03:26:24.341223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:24.340807    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h" event={"ID":"1d00176e-84de-4dd2-bc2e-42d663325e2a","Type":"ContainerDied","Data":"32e11c66300d6286658bb0cba9906aca65d8da2864054e044c090ee6a27f72eb"}
Oct 30 03:26:24.341223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:24.340829    5061 scope.go:122] "RemoveContainer" containerID="d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3"
Oct 30 03:26:24.351051 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:24.351035    5061 scope.go:122] "RemoveContainer" containerID="d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3"
Oct 30 03:26:24.351318 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:26:24.351297    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3\": container with ID starting with d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3 not found: ID does not exist" containerID="d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3"
Oct 30 03:26:24.351373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:24.351326    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3"} err="failed to get container status \"d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3\": rpc error: code = NotFound desc = could not find container \"d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3\": container with ID starting with d93b4913c08456fa6377335db31d145fb8366b8e93883dfd2764a2b135d47cb3 not found: ID does not exist"
Oct 30 03:26:24.390615 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:24.390593    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"]
Oct 30 03:26:24.400724 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:24.400705    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-723ffc-84c8887d64-nsq4h"]
Oct 30 03:26:24.702106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:24.702012    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:24.849642 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:24.849611    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d00176e-84de-4dd2-bc2e-42d663325e2a" path="/var/lib/kubelet/pods/1d00176e-84de-4dd2-bc2e-42d663325e2a/volumes"
Oct 30 03:26:25.110751 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.110712    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"]
Oct 30 03:26:25.111452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111422    5061 cpu_manager.go:414] "containerMap: removing container" podUID="aae61a65-0a49-4b8b-935f-45bb6981e8ba" containerName="payload-pre-processing"
Oct 30 03:26:25.111452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111445    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae61a65-0a49-4b8b-935f-45bb6981e8ba" containerName="payload-pre-processing"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111470    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1bce83b4-dca3-4ad5-a05e-940006d42ac3" containerName="istio-proxy"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111476    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bce83b4-dca3-4ad5-a05e-940006d42ac3" containerName="istio-proxy"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111486    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b997ff8f-09d5-44e8-8c66-3c89ff54b04f" containerName="payload-processing"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111493    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b997ff8f-09d5-44e8-8c66-3c89ff54b04f" containerName="payload-processing"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111508    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0166ec2f-c5c8-4387-91a7-15093053f4c4" containerName="maas-api"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111514    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0166ec2f-c5c8-4387-91a7-15093053f4c4" containerName="maas-api"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111524    5061 cpu_manager.go:414] "containerMap: removing container" podUID="83428494-94f2-427a-a101-2cc2d4f45a24" containerName="payload-processing"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111529    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83428494-94f2-427a-a101-2cc2d4f45a24" containerName="payload-processing"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111543    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111547    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111553    5061 cpu_manager.go:414] "containerMap: removing container" podUID="1d00176e-84de-4dd2-bc2e-42d663325e2a" containerName="payload-processing"
Oct 30 03:26:25.111595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111558    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d00176e-84de-4dd2-bc2e-42d663325e2a" containerName="payload-processing"
Oct 30 03:26:25.111979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111675    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bce83b4-dca3-4ad5-a05e-940006d42ac3" containerName="istio-proxy"
Oct 30 03:26:25.111979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111686    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b997ff8f-09d5-44e8-8c66-3c89ff54b04f" containerName="payload-processing"
Oct 30 03:26:25.111979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111693    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0166ec2f-c5c8-4387-91a7-15093053f4c4" containerName="maas-api"
Oct 30 03:26:25.111979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111700    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="83428494-94f2-427a-a101-2cc2d4f45a24" containerName="payload-processing"
Oct 30 03:26:25.111979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111707    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8eec092-64ae-43a2-8c3a-7c1e0f27b168" containerName="istio-proxy"
Oct 30 03:26:25.111979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111713    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d00176e-84de-4dd2-bc2e-42d663325e2a" containerName="payload-processing"
Oct 30 03:26:25.111979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.111725    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae61a65-0a49-4b8b-935f-45bb6981e8ba" containerName="payload-pre-processing"
Oct 30 03:26:25.116407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.116387    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.139841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.139807    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"]
Oct 30 03:26:25.147133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.147107    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-8b0787\"/\"rate-test-model-8b0787-kserve-self-signed-certs\""
Oct 30 03:26:25.147250 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.147193    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-8b0787\"/\"default-dockercfg-qcscb\""
Oct 30 03:26:25.155120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.155101    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-8b0787\"/\"kube-root-ca.crt\""
Oct 30 03:26:25.155214 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.155115    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-8b0787\"/\"openshift-service-ca.crt\""
Oct 30 03:26:25.190387 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.190360    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-dshm\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.190505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.190405    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c8b1046-7371-4f95-a724-532110f24a62-tls-certs\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.190505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.190470    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xs9d\" (UniqueName: \"kubernetes.io/projected/8c8b1046-7371-4f95-a724-532110f24a62-kube-api-access-6xs9d\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.190595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.190524    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-tmp-dir\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.190595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.190557    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-home\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.190717 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.190695    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-model-cache\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.198938 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.198911    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:25.291664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.291622    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-tmp-dir\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.291905 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.291683    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-home\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.291905 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.291798    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-model-cache\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.292046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.291938    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-dshm\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.292046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.291987    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c8b1046-7371-4f95-a724-532110f24a62-tls-certs\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.292046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.292026    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xs9d\" (UniqueName: \"kubernetes.io/projected/8c8b1046-7371-4f95-a724-532110f24a62-kube-api-access-6xs9d\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.292201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.292081    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-home\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.292201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.292137    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-tmp-dir\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.292201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.292173    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-model-cache\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.294298 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.294268    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-dshm\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.294519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.294502    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c8b1046-7371-4f95-a724-532110f24a62-tls-certs\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.313237 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.313210    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xs9d\" (UniqueName: \"kubernetes.io/projected/8c8b1046-7371-4f95-a724-532110f24a62-kube-api-access-6xs9d\") pod \"rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") " pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.430416 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.430323    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:25.447748 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:26:25.447714    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:26:25.735259 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:25.735232    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"]
Oct 30 03:26:25.736889 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:26:25.736859    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c8b1046_7371_4f95_a724_532110f24a62.slice/crio-31236a46c0ccbcf8f8e176a285e7dcb974f22ac1c284d56d67b8450f6cc8511c WatchSource:0}: Error finding container 31236a46c0ccbcf8f8e176a285e7dcb974f22ac1c284d56d67b8450f6cc8511c: Status 404 returned error can't find the container with id 31236a46c0ccbcf8f8e176a285e7dcb974f22ac1c284d56d67b8450f6cc8511c
Oct 30 03:26:26.371744 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:26.371713    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8" event={"ID":"8c8b1046-7371-4f95-a724-532110f24a62","Type":"ContainerStarted","Data":"bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8"}
Oct 30 03:26:26.371939 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:26.371752    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8" event={"ID":"8c8b1046-7371-4f95-a724-532110f24a62","Type":"ContainerStarted","Data":"31236a46c0ccbcf8f8e176a285e7dcb974f22ac1c284d56d67b8450f6cc8511c"}
Oct 30 03:26:26.411243 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:26.411193    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8" podStartSLOduration=1.4111797 podStartE2EDuration="1.4111797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:26.407652289 +0000 UTC m=+2731.709830656" watchObservedRunningTime="2026-10-30 03:26:26.4111797 +0000 UTC m=+2731.713358065"
Oct 30 03:26:26.431154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:26.431126    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:26.453791 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:26.453756    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:27.997430 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:27.997390    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:29.692463 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:29.692433    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:31.142121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:31.142086    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:32.778493 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.778457    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:32.884897 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.884860    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"]
Oct 30 03:26:32.889456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.889436    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:32.903473 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.903455    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-4b5459-openshift-default-dockercfg-4h4xx\""
Oct 30 03:26:32.920080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.920054    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"]
Oct 30 03:26:32.989178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.989145    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-data\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:32.989307 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.989182    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-certs\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:32.989307 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.989267    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-token\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:32.989307 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.989304    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-credential-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:32.989433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.989395    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-podinfo\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:32.989489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.989438    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:32.989584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.989566    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-envoy\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:32.989685 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.989670    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05c5ab7-4d93-4f89-8207-d440fffedf22-istiod-ca-cert\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:32.989754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:32.989738    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlrqv\" (UniqueName: \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-kube-api-access-wlrqv\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.090605 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.090575    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05c5ab7-4d93-4f89-8207-d440fffedf22-istiod-ca-cert\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.090765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.090640    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrqv\" (UniqueName: \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-kube-api-access-wlrqv\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.090765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.090675    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-data\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.090765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.090701    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-certs\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.090765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.090733    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-token\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.090765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.090762    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-credential-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.091065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.090816    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-podinfo\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.091065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.090879    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.091065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.090968    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-envoy\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.091480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.091440    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05c5ab7-4d93-4f89-8207-d440fffedf22-istiod-ca-cert\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.091801 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.091774    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-data\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.091951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.091813    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-certs\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.091951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.091906    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-credential-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.092232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.092203    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-socket\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.093839 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.093813    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-envoy\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.094080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.094059    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-podinfo\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.109655 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.109633    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-token\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.110163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.110140    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlrqv\" (UniqueName: \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-kube-api-access-wlrqv\") pod \"e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") " pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.203418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.203395    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:33.520017 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.519776    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"]
Oct 30 03:26:33.523031 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:26:33.522996    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05c5ab7_4d93_4f89_8207_d440fffedf22.slice/crio-86f17ab63ddf37cdca791997f94a89317c978480fead34cd901257035aa6432e WatchSource:0}: Error finding container 86f17ab63ddf37cdca791997f94a89317c978480fead34cd901257035aa6432e: Status 404 returned error can't find the container with id 86f17ab63ddf37cdca791997f94a89317c978480fead34cd901257035aa6432e
Oct 30 03:26:33.525568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.525535    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:33.525653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.525603    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:33.525653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.525630    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:33.821278 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.821234    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:33.974658 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:33.974627    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:34.209208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.209121    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"]
Oct 30 03:26:34.213229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.213201    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:26:34.238567 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.238542    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e9528f\""
Oct 30 03:26:34.238567 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.238561    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e9528f-dockercfg-g2lbg\""
Oct 30 03:26:34.242362 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.242337    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"]
Oct 30 03:26:34.327529 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.327499    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8b1b1e-db85-4dde-942b-3cb7920a4738-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b\" (UID: \"0d8b1b1e-db85-4dde-942b-3cb7920a4738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:26:34.327707 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.327550    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8bf8\" (UniqueName: \"kubernetes.io/projected/0d8b1b1e-db85-4dde-942b-3cb7920a4738-kube-api-access-r8bf8\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b\" (UID: \"0d8b1b1e-db85-4dde-942b-3cb7920a4738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:26:34.384974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.384946    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"]
Oct 30 03:26:34.389174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.389158    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:26:34.407899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.407876    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"]
Oct 30 03:26:34.426083 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.426054    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" event={"ID":"d05c5ab7-4d93-4f89-8207-d440fffedf22","Type":"ContainerStarted","Data":"a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d"}
Oct 30 03:26:34.426228 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.426095    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" event={"ID":"d05c5ab7-4d93-4f89-8207-d440fffedf22","Type":"ContainerStarted","Data":"86f17ab63ddf37cdca791997f94a89317c978480fead34cd901257035aa6432e"}
Oct 30 03:26:34.429422 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.429397    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96459a9f-427b-42d4-a568-d0a3cc7c9243-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4\" (UID: \"96459a9f-427b-42d4-a568-d0a3cc7c9243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:26:34.429544 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.429464    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8b1b1e-db85-4dde-942b-3cb7920a4738-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b\" (UID: \"0d8b1b1e-db85-4dde-942b-3cb7920a4738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:26:34.429622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.429566    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8bf8\" (UniqueName: \"kubernetes.io/projected/0d8b1b1e-db85-4dde-942b-3cb7920a4738-kube-api-access-r8bf8\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b\" (UID: \"0d8b1b1e-db85-4dde-942b-3cb7920a4738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:26:34.429688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.429670    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdjnf\" (UniqueName: \"kubernetes.io/projected/96459a9f-427b-42d4-a568-d0a3cc7c9243-kube-api-access-sdjnf\") pod \"payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4\" (UID: \"96459a9f-427b-42d4-a568-d0a3cc7c9243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:26:34.430209 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.430188    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8b1b1e-db85-4dde-942b-3cb7920a4738-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b\" (UID: \"0d8b1b1e-db85-4dde-942b-3cb7920a4738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:26:34.448214 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.448188    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8bf8\" (UniqueName: \"kubernetes.io/projected/0d8b1b1e-db85-4dde-942b-3cb7920a4738-kube-api-access-r8bf8\") pod \"payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b\" (UID: \"0d8b1b1e-db85-4dde-942b-3cb7920a4738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:26:34.467167 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.467075    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" podStartSLOduration=2.467060134 podStartE2EDuration="2.467060134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:34.46375336 +0000 UTC m=+2739.765931724" watchObservedRunningTime="2026-10-30 03:26:34.467060134 +0000 UTC m=+2739.769238499"
Oct 30 03:26:34.547887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.543955    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96459a9f-427b-42d4-a568-d0a3cc7c9243-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4\" (UID: \"96459a9f-427b-42d4-a568-d0a3cc7c9243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:26:34.547887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.544224    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdjnf\" (UniqueName: \"kubernetes.io/projected/96459a9f-427b-42d4-a568-d0a3cc7c9243-kube-api-access-sdjnf\") pod \"payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4\" (UID: \"96459a9f-427b-42d4-a568-d0a3cc7c9243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:26:34.547887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.544837    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:26:34.547887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.546404    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96459a9f-427b-42d4-a568-d0a3cc7c9243-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4\" (UID: \"96459a9f-427b-42d4-a568-d0a3cc7c9243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:26:34.571892 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.566604    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdjnf\" (UniqueName: \"kubernetes.io/projected/96459a9f-427b-42d4-a568-d0a3cc7c9243-kube-api-access-sdjnf\") pod \"payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4\" (UID: \"96459a9f-427b-42d4-a568-d0a3cc7c9243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:26:34.612909 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.612874    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:34.703302 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.703260    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:26:34.902663 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:34.902623    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"]
Oct 30 03:26:35.108842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.108818    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"]
Oct 30 03:26:35.204301 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.204093    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:35.205390 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.205366    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 03:26:35.205506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.205412    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" podUID="d05c5ab7-4d93-4f89-8207-d440fffedf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 03:26:35.253915 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.252472    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:35.374077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.374039    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:35.456392 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.455332    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:35.484221 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.484184    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4" event={"ID":"96459a9f-427b-42d4-a568-d0a3cc7c9243","Type":"ContainerStarted","Data":"0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641"}
Oct 30 03:26:35.484486 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.484468    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4" event={"ID":"96459a9f-427b-42d4-a568-d0a3cc7c9243","Type":"ContainerStarted","Data":"9820f51f45b06b9ae0fe19303bbb0cabdda799bc9593a65f020267b5e766e595"}
Oct 30 03:26:35.484748 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.484724    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:26:35.487584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.487563    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b" event={"ID":"0d8b1b1e-db85-4dde-942b-3cb7920a4738","Type":"ContainerStarted","Data":"00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b"}
Oct 30 03:26:35.487726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.487715    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b" event={"ID":"0d8b1b1e-db85-4dde-942b-3cb7920a4738","Type":"ContainerStarted","Data":"592404dac08d7ee985584461bd3d896a20860dfa82da8dfcef476c1a1fef60ec"}
Oct 30 03:26:35.487889 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.487879    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:26:35.495106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.495082    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:35.512746 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.512684    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:35.540871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.532174    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4" podStartSLOduration=1.5321567630000001 podStartE2EDuration="1.532156763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:35.51844577 +0000 UTC m=+2740.820624147" watchObservedRunningTime="2026-10-30 03:26:35.532156763 +0000 UTC m=+2740.834335131"
Oct 30 03:26:35.594465 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.594413    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b" podStartSLOduration=1.5944003869999999 podStartE2EDuration="1.594400387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:35.592139039 +0000 UTC m=+2740.894317405" watchObservedRunningTime="2026-10-30 03:26:35.594400387 +0000 UTC m=+2740.896578749"
Oct 30 03:26:35.842892 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:35.841987    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:36.204376 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:36.204302    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 03:26:36.204376 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:36.204352    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" podUID="d05c5ab7-4d93-4f89-8207-d440fffedf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 03:26:36.460067 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:36.459981    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:26:37.074028 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:37.073992    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:37.204534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:37.204502    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 03:26:37.204993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:37.204562    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" podUID="d05c5ab7-4d93-4f89-8207-d440fffedf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 03:26:38.204059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:38.204023    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 03:26:38.204248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:38.204086    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" podUID="d05c5ab7-4d93-4f89-8207-d440fffedf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 03:26:38.572560 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:38.572520    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:39.203699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:39.203664    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 03:26:39.203915 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:39.203727    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" podUID="d05c5ab7-4d93-4f89-8207-d440fffedf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 03:26:40.074189 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:40.074143    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:40.204550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:40.204506    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 03:26:40.204726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:40.204568    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" podUID="d05c5ab7-4d93-4f89-8207-d440fffedf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 03:26:41.210170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:41.210134    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:41.210570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:41.210526    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:41.211245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:41.211225    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:26:41.800812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:41.800773    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:42.067797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:42.067723    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:42.203753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:42.203720    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:42.366903 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:42.366781    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"]
Oct 30 03:26:42.367380 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:42.367225    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy" containerID="cri-o://e474a967146a61644872461d96251e2511c2580a14297bc92434e5f4e065e0e4" gracePeriod=30
Oct 30 03:26:42.413764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:42.413732    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:42.573223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:42.573183    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:43.571741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:43.571710    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:44.993902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:44.993870    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:45.179264 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:45.179235    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:45.324782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:45.324745    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:45.537785 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:45.537750    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:45.861629 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:45.861588    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:46.224984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:46.224914    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:46.387776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:46.387738    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:46.496624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:46.496542    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:26:46.567497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:46.567464    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:46.949339 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:46.949306    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:26:46.949514 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:46.949365    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:26:47.502045 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.502009    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:26:47.594820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.594782    5061 generic.go:356] "Generic (PLEG): container finished" podID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerID="e474a967146a61644872461d96251e2511c2580a14297bc92434e5f4e065e0e4" exitCode=0
Oct 30 03:26:47.595042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.594835    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" event={"ID":"f6dc0d03-1186-414b-b702-aa748cd029d4","Type":"ContainerDied","Data":"e474a967146a61644872461d96251e2511c2580a14297bc92434e5f4e065e0e4"}
Oct 30 03:26:47.595042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.594895    5061 scope.go:122] "RemoveContainer" containerID="8b5f98dd709733de2400bc68ef15930920f81aa10e5e6294691b46231ca1f27d"
Oct 30 03:26:47.656812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.656784    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:26:47.682525 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.682500    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:47.713744 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.713706    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-kube-api-access-lpqh8\" (UniqueName: \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-kube-api-access-lpqh8\") pod \"f6dc0d03-1186-414b-b702-aa748cd029d4\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") "
Oct 30 03:26:47.714128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.713903    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6dc0d03-1186-414b-b702-aa748cd029d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6dc0d03-1186-414b-b702-aa748cd029d4-istiod-ca-cert\") pod \"f6dc0d03-1186-414b-b702-aa748cd029d4\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") "
Oct 30 03:26:47.714128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.713999    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-envoy\") pod \"f6dc0d03-1186-414b-b702-aa748cd029d4\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") "
Oct 30 03:26:47.714128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.714052    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-podinfo\") pod \"f6dc0d03-1186-414b-b702-aa748cd029d4\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") "
Oct 30 03:26:47.714128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.714090    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-token\" (UniqueName: \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-token\") pod \"f6dc0d03-1186-414b-b702-aa748cd029d4\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") "
Oct 30 03:26:47.714493 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.714139    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-socket\") pod \"f6dc0d03-1186-414b-b702-aa748cd029d4\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") "
Oct 30 03:26:47.714493 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.714187    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-credential-socket\") pod \"f6dc0d03-1186-414b-b702-aa748cd029d4\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") "
Oct 30 03:26:47.714493 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.714218    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-data\") pod \"f6dc0d03-1186-414b-b702-aa748cd029d4\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") "
Oct 30 03:26:47.714493 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.714309    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-certs\") pod \"f6dc0d03-1186-414b-b702-aa748cd029d4\" (UID: \"f6dc0d03-1186-414b-b702-aa748cd029d4\") "
Oct 30 03:26:47.715247 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.715219    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-certs" pod "f6dc0d03-1186-414b-b702-aa748cd029d4" (UID: "f6dc0d03-1186-414b-b702-aa748cd029d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:47.720980 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.717739    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-kube-api-access-lpqh8" pod "f6dc0d03-1186-414b-b702-aa748cd029d4" (UID: "f6dc0d03-1186-414b-b702-aa748cd029d4"). InnerVolumeSpecName "kube-api-access-lpqh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:47.720980 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.718143    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-podinfo" pod "f6dc0d03-1186-414b-b702-aa748cd029d4" (UID: "f6dc0d03-1186-414b-b702-aa748cd029d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:26:47.721877 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.721535    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6dc0d03-1186-414b-b702-aa748cd029d4-istiod-ca-cert" pod "f6dc0d03-1186-414b-b702-aa748cd029d4" (UID: "f6dc0d03-1186-414b-b702-aa748cd029d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:47.721877 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.721799    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-socket" pod "f6dc0d03-1186-414b-b702-aa748cd029d4" (UID: "f6dc0d03-1186-414b-b702-aa748cd029d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:47.722059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.722038    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-credential-socket" pod "f6dc0d03-1186-414b-b702-aa748cd029d4" (UID: "f6dc0d03-1186-414b-b702-aa748cd029d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:47.722220 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.722132    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-data" pod "f6dc0d03-1186-414b-b702-aa748cd029d4" (UID: "f6dc0d03-1186-414b-b702-aa748cd029d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:47.723090 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.723061    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-token" pod "f6dc0d03-1186-414b-b702-aa748cd029d4" (UID: "f6dc0d03-1186-414b-b702-aa748cd029d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:47.723878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.723819    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-envoy" pod "f6dc0d03-1186-414b-b702-aa748cd029d4" (UID: "f6dc0d03-1186-414b-b702-aa748cd029d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:47.815947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.815918    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:47.815947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.815943    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:47.815947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.815953    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:47.816163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.815962    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:47.816163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.815973    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:47.816163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.815982    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:47.816163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.815991    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6dc0d03-1186-414b-b702-aa748cd029d4-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:47.816163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.815999    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpqh8\" (UniqueName: \"kubernetes.io/projected/f6dc0d03-1186-414b-b702-aa748cd029d4-kube-api-access-lpqh8\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:47.816163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:47.816007    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6dc0d03-1186-414b-b702-aa748cd029d4-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:48.603378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:48.603350    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"
Oct 30 03:26:48.603819 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:48.603385    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp" event={"ID":"f6dc0d03-1186-414b-b702-aa748cd029d4","Type":"ContainerDied","Data":"d281e76aff7ae6daf10c4c66460eb0c02feda28128d5281e93e20216632c4d5c"}
Oct 30 03:26:48.603819 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:48.603438    5061 scope.go:122] "RemoveContainer" containerID="e474a967146a61644872461d96251e2511c2580a14297bc92434e5f4e065e0e4"
Oct 30 03:26:48.663031 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:48.663009    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"]
Oct 30 03:26:48.673686 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:48.673658    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-723ffc-openshift-default-5b57c844cf-cnsjp"]
Oct 30 03:26:48.858477 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:48.858399    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" path="/var/lib/kubelet/pods/f6dc0d03-1186-414b-b702-aa748cd029d4/volumes"
Oct 30 03:26:49.287577 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:49.287497    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:50.286591 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:50.286562    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:50.703560 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:50.703529    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:51.057490 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:51.057460    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:51.400715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:51.400634    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:51.570584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:51.570552    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:52.875234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:52.875197    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:53.003080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:53.003040    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:53.333426 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:53.333393    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:53.394906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:53.394862    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"]
Oct 30 03:26:53.428182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:53.428141    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"]
Oct 30 03:26:53.578228 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:53.578192    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:53.714047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:53.713963    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:54.686876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:54.686822    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:54.854980 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:54.854946    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:55.449354 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:26:55.449319    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:26:55.952196 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:55.952156    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:56.629926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:56.627332    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:57.192466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:57.192427    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:57.421691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:57.421662    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:57.587920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:57.587887    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:58.275095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.275060    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:58.395664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.395625    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4" podUID="f8bb1f50-839d-44e1-9d5a-9b0401f324c0" containerName="payload-processing" containerID="cri-o://faa01350fc7e59763c1f9d05cd6b9af30e28b2b4f58710ad2bd6ed3d875143cc" gracePeriod=25
Oct 30 03:26:58.412096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.412071    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"]
Oct 30 03:26:58.412690 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.412677    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy"
Oct 30 03:26:58.412736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.412691    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy"
Oct 30 03:26:58.412736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.412702    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy"
Oct 30 03:26:58.412736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.412707    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy"
Oct 30 03:26:58.412939 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.412804    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy"
Oct 30 03:26:58.416438 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.416419    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:26:58.422229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.422204    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:58.429705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.429680    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt" podUID="7b5fe7e6-dd7b-4386-8504-ea4507d6541e" containerName="payload-pre-processing" containerID="cri-o://63eb29e7ca2230f607870f6ad125877f6b91535386ae4c8f612eef5f9fc8072f" gracePeriod=25
Oct 30 03:26:58.435836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.435815    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"]
Oct 30 03:26:58.443154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.443119    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-055daa\""
Oct 30 03:26:58.443154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.443145    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-055daa-dockercfg-rscpp\""
Oct 30 03:26:58.512872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.511796    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmxjp\" (UniqueName: \"kubernetes.io/projected/00539a57-b3bc-412c-b30c-166adc0a3844-kube-api-access-lmxjp\") pod \"payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5\" (UID: \"00539a57-b3bc-412c-b30c-166adc0a3844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:26:58.512872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.511985    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00539a57-b3bc-412c-b30c-166adc0a3844-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5\" (UID: \"00539a57-b3bc-412c-b30c-166adc0a3844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:26:58.615619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.615563    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00539a57-b3bc-412c-b30c-166adc0a3844-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5\" (UID: \"00539a57-b3bc-412c-b30c-166adc0a3844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:26:58.615816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.615798    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmxjp\" (UniqueName: \"kubernetes.io/projected/00539a57-b3bc-412c-b30c-166adc0a3844-kube-api-access-lmxjp\") pod \"payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5\" (UID: \"00539a57-b3bc-412c-b30c-166adc0a3844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:26:58.616828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.616795    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00539a57-b3bc-412c-b30c-166adc0a3844-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5\" (UID: \"00539a57-b3bc-412c-b30c-166adc0a3844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:26:58.635752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.634589    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmxjp\" (UniqueName: \"kubernetes.io/projected/00539a57-b3bc-412c-b30c-166adc0a3844-kube-api-access-lmxjp\") pod \"payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5\" (UID: \"00539a57-b3bc-412c-b30c-166adc0a3844\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:26:58.678675 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.678644    5061 generic.go:356] "Generic (PLEG): container finished" podID="7b5fe7e6-dd7b-4386-8504-ea4507d6541e" containerID="63eb29e7ca2230f607870f6ad125877f6b91535386ae4c8f612eef5f9fc8072f" exitCode=0
Oct 30 03:26:58.678870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.678723    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt" event={"ID":"7b5fe7e6-dd7b-4386-8504-ea4507d6541e","Type":"ContainerDied","Data":"63eb29e7ca2230f607870f6ad125877f6b91535386ae4c8f612eef5f9fc8072f"}
Oct 30 03:26:58.680914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.680884    5061 generic.go:356] "Generic (PLEG): container finished" podID="f8bb1f50-839d-44e1-9d5a-9b0401f324c0" containerID="faa01350fc7e59763c1f9d05cd6b9af30e28b2b4f58710ad2bd6ed3d875143cc" exitCode=0
Oct 30 03:26:58.681037 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.680993    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4" event={"ID":"f8bb1f50-839d-44e1-9d5a-9b0401f324c0","Type":"ContainerDied","Data":"faa01350fc7e59763c1f9d05cd6b9af30e28b2b4f58710ad2bd6ed3d875143cc"}
Oct 30 03:26:58.770952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.770921    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:26:58.801709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.801653    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:26:58.896903 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.896831    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:26:58.922616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.922595    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-plugins-config-volume\") pod \"f8bb1f50-839d-44e1-9d5a-9b0401f324c0\" (UID: \"f8bb1f50-839d-44e1-9d5a-9b0401f324c0\") "
Oct 30 03:26:58.922786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.922771    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-kube-api-access-clmng\" (UniqueName: \"kubernetes.io/projected/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-kube-api-access-clmng\") pod \"7b5fe7e6-dd7b-4386-8504-ea4507d6541e\" (UID: \"7b5fe7e6-dd7b-4386-8504-ea4507d6541e\") "
Oct 30 03:26:58.922908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.922826    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-plugins-config-volume\") pod \"7b5fe7e6-dd7b-4386-8504-ea4507d6541e\" (UID: \"7b5fe7e6-dd7b-4386-8504-ea4507d6541e\") "
Oct 30 03:26:58.923012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.922997    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-kube-api-access-grhw5\" (UniqueName: \"kubernetes.io/projected/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-kube-api-access-grhw5\") pod \"f8bb1f50-839d-44e1-9d5a-9b0401f324c0\" (UID: \"f8bb1f50-839d-44e1-9d5a-9b0401f324c0\") "
Oct 30 03:26:58.923076 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.923008    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-plugins-config-volume" pod "f8bb1f50-839d-44e1-9d5a-9b0401f324c0" (UID: "f8bb1f50-839d-44e1-9d5a-9b0401f324c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:58.923356 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.923322    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-plugins-config-volume" pod "7b5fe7e6-dd7b-4386-8504-ea4507d6541e" (UID: "7b5fe7e6-dd7b-4386-8504-ea4507d6541e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:58.923569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.923547    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:58.923644 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.923577    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:58.924998 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.924979    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-kube-api-access-grhw5" pod "f8bb1f50-839d-44e1-9d5a-9b0401f324c0" (UID: "f8bb1f50-839d-44e1-9d5a-9b0401f324c0"). InnerVolumeSpecName "kube-api-access-grhw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:58.925269 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:58.925249    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-kube-api-access-clmng" pod "7b5fe7e6-dd7b-4386-8504-ea4507d6541e" (UID: "7b5fe7e6-dd7b-4386-8504-ea4507d6541e"). InnerVolumeSpecName "kube-api-access-clmng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:59.025629 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.025567    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clmng\" (UniqueName: \"kubernetes.io/projected/7b5fe7e6-dd7b-4386-8504-ea4507d6541e-kube-api-access-clmng\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:59.025629 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.025602    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grhw5\" (UniqueName: \"kubernetes.io/projected/f8bb1f50-839d-44e1-9d5a-9b0401f324c0-kube-api-access-grhw5\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:26:59.200970 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.200827    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"]
Oct 30 03:26:59.204240 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:26:59.204214    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00539a57_b3bc_412c_b30c_166adc0a3844.slice/crio-178d019e120882131337fb76e0892330d4411f82517cf0a9bf3dcf626bebd89e WatchSource:0}: Error finding container 178d019e120882131337fb76e0892330d4411f82517cf0a9bf3dcf626bebd89e: Status 404 returned error can't find the container with id 178d019e120882131337fb76e0892330d4411f82517cf0a9bf3dcf626bebd89e
Oct 30 03:26:59.582564 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.582532    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:59.688217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.688189    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"
Oct 30 03:26:59.688394 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.688214    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4" event={"ID":"f8bb1f50-839d-44e1-9d5a-9b0401f324c0","Type":"ContainerDied","Data":"ee638e83e28c86ee0ad0ba415af3be93745c30314c32a65fb9f2ad418bd4fe43"}
Oct 30 03:26:59.688394 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.688284    5061 scope.go:122] "RemoveContainer" containerID="faa01350fc7e59763c1f9d05cd6b9af30e28b2b4f58710ad2bd6ed3d875143cc"
Oct 30 03:26:59.689900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.689874    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5" event={"ID":"00539a57-b3bc-412c-b30c-166adc0a3844","Type":"ContainerStarted","Data":"06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895"}
Oct 30 03:26:59.690010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.689929    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:26:59.690010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.689950    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5" event={"ID":"00539a57-b3bc-412c-b30c-166adc0a3844","Type":"ContainerStarted","Data":"178d019e120882131337fb76e0892330d4411f82517cf0a9bf3dcf626bebd89e"}
Oct 30 03:26:59.691374 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.691356    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"
Oct 30 03:26:59.691475 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.691391    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt" event={"ID":"7b5fe7e6-dd7b-4386-8504-ea4507d6541e","Type":"ContainerDied","Data":"3c2890fff0c60634bf8d162cda855219708115a1ba4ea58dd3ca3b416c98f65c"}
Oct 30 03:26:59.700264 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.700247    5061 scope.go:122] "RemoveContainer" containerID="63eb29e7ca2230f607870f6ad125877f6b91535386ae4c8f612eef5f9fc8072f"
Oct 30 03:26:59.747451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.747424    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"]
Oct 30 03:26:59.751820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.751799    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e2727-5dbb46544b-5knb4"]
Oct 30 03:26:59.786337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.786297    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5" podStartSLOduration=1.786285302 podStartE2EDuration="1.786285302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:59.785718491 +0000 UTC m=+2765.087896857" watchObservedRunningTime="2026-10-30 03:26:59.786285302 +0000 UTC m=+2765.088463727"
Oct 30 03:26:59.818701 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.818678    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"]
Oct 30 03:26:59.826423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:26:59.826395    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e2727-747c8cc64c-lhsqt"]
Oct 30 03:27:00.818248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:00.818210    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:00.848093 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:00.848061    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b5fe7e6-dd7b-4386-8504-ea4507d6541e" path="/var/lib/kubelet/pods/7b5fe7e6-dd7b-4386-8504-ea4507d6541e/volumes"
Oct 30 03:27:00.848597 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:00.848575    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8bb1f50-839d-44e1-9d5a-9b0401f324c0" path="/var/lib/kubelet/pods/f8bb1f50-839d-44e1-9d5a-9b0401f324c0/volumes"
Oct 30 03:27:01.057015 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:01.056969    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:01.821437 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:01.821374    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:02.556802 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.556751    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:02.686433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.686400    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"]
Oct 30 03:27:02.687011 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.686997    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f8bb1f50-839d-44e1-9d5a-9b0401f324c0" containerName="payload-processing"
Oct 30 03:27:02.687058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.687012    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8bb1f50-839d-44e1-9d5a-9b0401f324c0" containerName="payload-processing"
Oct 30 03:27:02.687058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.687038    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7b5fe7e6-dd7b-4386-8504-ea4507d6541e" containerName="payload-pre-processing"
Oct 30 03:27:02.687058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.687043    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5fe7e6-dd7b-4386-8504-ea4507d6541e" containerName="payload-pre-processing"
Oct 30 03:27:02.687174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.687148    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8bb1f50-839d-44e1-9d5a-9b0401f324c0" containerName="payload-processing"
Oct 30 03:27:02.687174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.687161    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5fe7e6-dd7b-4386-8504-ea4507d6541e" containerName="payload-pre-processing"
Oct 30 03:27:02.687174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.687168    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6dc0d03-1186-414b-b702-aa748cd029d4" containerName="istio-proxy"
Oct 30 03:27:02.691629 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.691614    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:27:02.708254 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.708232    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cf605a-dockercfg-kbxhj\""
Oct 30 03:27:02.718927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.718902    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cf605a\""
Oct 30 03:27:02.722484 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.722461    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"]
Oct 30 03:27:02.732703 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.732674    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"]
Oct 30 03:27:02.737743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.737723    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:27:02.773090 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.768082    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:02.796872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.791645    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmznj\" (UniqueName: \"kubernetes.io/projected/650d875b-fa01-4dee-89a5-f83cabfd99a2-kube-api-access-kmznj\") pod \"payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6\" (UID: \"650d875b-fa01-4dee-89a5-f83cabfd99a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:27:02.796872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.791870    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650d875b-fa01-4dee-89a5-f83cabfd99a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6\" (UID: \"650d875b-fa01-4dee-89a5-f83cabfd99a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:27:02.796872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.791906    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh697\" (UniqueName: \"kubernetes.io/projected/a036f32a-965c-435a-ade5-8669e02fcb25-kube-api-access-jh697\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7\" (UID: \"a036f32a-965c-435a-ade5-8669e02fcb25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:27:02.796872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.791979    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a036f32a-965c-435a-ade5-8669e02fcb25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7\" (UID: \"a036f32a-965c-435a-ade5-8669e02fcb25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:27:02.854944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.818925    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"]
Oct 30 03:27:02.893726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.893693    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a036f32a-965c-435a-ade5-8669e02fcb25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7\" (UID: \"a036f32a-965c-435a-ade5-8669e02fcb25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:27:02.893920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.893785    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmznj\" (UniqueName: \"kubernetes.io/projected/650d875b-fa01-4dee-89a5-f83cabfd99a2-kube-api-access-kmznj\") pod \"payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6\" (UID: \"650d875b-fa01-4dee-89a5-f83cabfd99a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:27:02.894010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.893918    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650d875b-fa01-4dee-89a5-f83cabfd99a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6\" (UID: \"650d875b-fa01-4dee-89a5-f83cabfd99a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:27:02.894010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.893950    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh697\" (UniqueName: \"kubernetes.io/projected/a036f32a-965c-435a-ade5-8669e02fcb25-kube-api-access-jh697\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7\" (UID: \"a036f32a-965c-435a-ade5-8669e02fcb25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:27:02.894711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.894680    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a036f32a-965c-435a-ade5-8669e02fcb25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7\" (UID: \"a036f32a-965c-435a-ade5-8669e02fcb25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:27:02.894873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.894833    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650d875b-fa01-4dee-89a5-f83cabfd99a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6\" (UID: \"650d875b-fa01-4dee-89a5-f83cabfd99a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:27:02.911787 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.911759    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh697\" (UniqueName: \"kubernetes.io/projected/a036f32a-965c-435a-ade5-8669e02fcb25-kube-api-access-jh697\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7\" (UID: \"a036f32a-965c-435a-ade5-8669e02fcb25\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:27:02.911933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:02.911839    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmznj\" (UniqueName: \"kubernetes.io/projected/650d875b-fa01-4dee-89a5-f83cabfd99a2-kube-api-access-kmznj\") pod \"payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6\" (UID: \"650d875b-fa01-4dee-89a5-f83cabfd99a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:27:03.005141 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.005110    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:27:03.070140 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.069189    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:27:03.306743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.306674    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"]
Oct 30 03:27:03.309320 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:27:03.309296    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda036f32a_965c_435a_ade5_8669e02fcb25.slice/crio-d9c33d7531487a863ce864c84a4ef8996489943d76ec85d99447b11c0cacdfb6 WatchSource:0}: Error finding container d9c33d7531487a863ce864c84a4ef8996489943d76ec85d99447b11c0cacdfb6: Status 404 returned error can't find the container with id d9c33d7531487a863ce864c84a4ef8996489943d76ec85d99447b11c0cacdfb6
Oct 30 03:27:03.384419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.384391    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"]
Oct 30 03:27:03.387766 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:27:03.387742    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650d875b_fa01_4dee_89a5_f83cabfd99a2.slice/crio-6e1d7434e7066e5aa3dfdc5452bd8959f9f80f69286c8893e236ca7c5671f408 WatchSource:0}: Error finding container 6e1d7434e7066e5aa3dfdc5452bd8959f9f80f69286c8893e236ca7c5671f408: Status 404 returned error can't find the container with id 6e1d7434e7066e5aa3dfdc5452bd8959f9f80f69286c8893e236ca7c5671f408
Oct 30 03:27:03.726820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.726718    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7" event={"ID":"a036f32a-965c-435a-ade5-8669e02fcb25","Type":"ContainerStarted","Data":"35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20"}
Oct 30 03:27:03.726820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.726776    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7" event={"ID":"a036f32a-965c-435a-ade5-8669e02fcb25","Type":"ContainerStarted","Data":"d9c33d7531487a863ce864c84a4ef8996489943d76ec85d99447b11c0cacdfb6"}
Oct 30 03:27:03.726820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.726807    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:27:03.728696 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.728666    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6" event={"ID":"650d875b-fa01-4dee-89a5-f83cabfd99a2","Type":"ContainerStarted","Data":"d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24"}
Oct 30 03:27:03.728861 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.728718    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6" event={"ID":"650d875b-fa01-4dee-89a5-f83cabfd99a2","Type":"ContainerStarted","Data":"6e1d7434e7066e5aa3dfdc5452bd8959f9f80f69286c8893e236ca7c5671f408"}
Oct 30 03:27:03.728938 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.728858    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:27:03.753157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.753134    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:03.762696 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.762655    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7" podStartSLOduration=1.76264377 podStartE2EDuration="1.76264377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:03.761946336 +0000 UTC m=+2769.064124701" watchObservedRunningTime="2026-10-30 03:27:03.76264377 +0000 UTC m=+2769.064822134"
Oct 30 03:27:03.800237 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:03.800194    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6" podStartSLOduration=1.8001825870000001 podStartE2EDuration="1.800182587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:03.797438289 +0000 UTC m=+2769.099616654" watchObservedRunningTime="2026-10-30 03:27:03.800182587 +0000 UTC m=+2769.102360951"
Oct 30 03:27:04.215232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:04.215198    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"]
Oct 30 03:27:04.379046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:04.378894    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:04.496269 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:04.496161    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:05.495696 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.495660    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"]
Oct 30 03:27:05.496110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.495838    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp" podUID="d440d2d2-15f6-4053-817f-19622b65948b" containerName="maas-api" containerID="cri-o://377ae59fd8f407d0bf2011c7ac9d4c98a08f9812af297a0887f5d590012b93ec" gracePeriod=30
Oct 30 03:27:05.581734 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.581705    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:05.790714 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.790683    5061 generic.go:356] "Generic (PLEG): container finished" podID="d440d2d2-15f6-4053-817f-19622b65948b" containerID="377ae59fd8f407d0bf2011c7ac9d4c98a08f9812af297a0887f5d590012b93ec" exitCode=0
Oct 30 03:27:05.790833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.790777    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp" event={"ID":"d440d2d2-15f6-4053-817f-19622b65948b","Type":"ContainerDied","Data":"377ae59fd8f407d0bf2011c7ac9d4c98a08f9812af297a0887f5d590012b93ec"}
Oct 30 03:27:05.814820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.814789    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:27:05.937872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.937164    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-maas-api-tls\") pod \"d440d2d2-15f6-4053-817f-19622b65948b\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") "
Oct 30 03:27:05.937872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.937252    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d440d2d2-15f6-4053-817f-19622b65948b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d440d2d2-15f6-4053-817f-19622b65948b-openshift-service-ca\") pod \"d440d2d2-15f6-4053-817f-19622b65948b\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") "
Oct 30 03:27:05.937872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.937303    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-metrics-certs\") pod \"d440d2d2-15f6-4053-817f-19622b65948b\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") "
Oct 30 03:27:05.937872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.937404    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d440d2d2-15f6-4053-817f-19622b65948b-kube-api-access-hqchm\" (UniqueName: \"kubernetes.io/projected/d440d2d2-15f6-4053-817f-19622b65948b-kube-api-access-hqchm\") pod \"d440d2d2-15f6-4053-817f-19622b65948b\" (UID: \"d440d2d2-15f6-4053-817f-19622b65948b\") "
Oct 30 03:27:05.939688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.938250    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d440d2d2-15f6-4053-817f-19622b65948b-openshift-service-ca" pod "d440d2d2-15f6-4053-817f-19622b65948b" (UID: "d440d2d2-15f6-4053-817f-19622b65948b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:05.948932 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.948187    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-metrics-certs" pod "d440d2d2-15f6-4053-817f-19622b65948b" (UID: "d440d2d2-15f6-4053-817f-19622b65948b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:05.948932 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.948327    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-maas-api-tls" pod "d440d2d2-15f6-4053-817f-19622b65948b" (UID: "d440d2d2-15f6-4053-817f-19622b65948b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:05.948932 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:05.948353    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d440d2d2-15f6-4053-817f-19622b65948b-kube-api-access-hqchm" pod "d440d2d2-15f6-4053-817f-19622b65948b" (UID: "d440d2d2-15f6-4053-817f-19622b65948b"). InnerVolumeSpecName "kube-api-access-hqchm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:06.039022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.038987    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:06.039022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.039016    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d440d2d2-15f6-4053-817f-19622b65948b-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:06.039022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.039026    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d440d2d2-15f6-4053-817f-19622b65948b-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:06.039251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.039034    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqchm\" (UniqueName: \"kubernetes.io/projected/d440d2d2-15f6-4053-817f-19622b65948b-kube-api-access-hqchm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:06.266251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.266218    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:06.412002 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.411968    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:06.532178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.532081    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:06.798906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.798865    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"
Oct 30 03:27:06.798906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.798880    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp" event={"ID":"d440d2d2-15f6-4053-817f-19622b65948b","Type":"ContainerDied","Data":"b9478454bcead948ebeac2a4333c6aa1693db113fd83f6046ba706c2ba092fe1"}
Oct 30 03:27:06.799144 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.798949    5061 scope.go:122] "RemoveContainer" containerID="377ae59fd8f407d0bf2011c7ac9d4c98a08f9812af297a0887f5d590012b93ec"
Oct 30 03:27:06.848477 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.848453    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"]
Oct 30 03:27:06.855497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.855476    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ac7936-6b5ff89b89-cvskp"]
Oct 30 03:27:06.898194 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:06.898171    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:08.185335 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:08.185289    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:08.847894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:08.847843    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d440d2d2-15f6-4053-817f-19622b65948b" path="/var/lib/kubelet/pods/d440d2d2-15f6-4053-817f-19622b65948b/volumes"
Oct 30 03:27:09.215962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:09.215840    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg" podUID="df3304e2-e0ca-482e-8f79-52e0724e4663" containerName="payload-processing" containerID="cri-o://f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635" gracePeriod=25
Oct 30 03:27:09.795464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:09.795421    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:10.704244 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:10.704213    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:27:11.053042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:11.053011    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:11.600892 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:11.600843    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:11.717595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:11.717564    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:12.179497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:12.179422    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:13.346143 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:13.346072    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:13.483293 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:13.483195    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:13.633383 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:13.633135    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:13.634991 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:13.634927    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"]
Oct 30 03:27:13.635704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:13.635652    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy" containerID="cri-o://530e418846cf60006385ff34622d6cf71651e9c25e822c681d5acf49c531522a" gracePeriod=30
Oct 30 03:27:13.848108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:13.848006    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:14.742178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:14.742109    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:27:14.743102 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:14.742273    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:27:15.271113 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:15.271036    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:16.595230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:16.595193    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:27:16.595645 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:16.595268    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:27:16.827338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:16.827305    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:17.749245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:17.749211    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg" podUID="df3304e2-e0ca-482e-8f79-52e0724e4663" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:18.165949 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.165918    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:18.283956 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.283915    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:18.517685 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.517594    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:18.652419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.652384    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:18.898384 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.898354    5061 generic.go:356] "Generic (PLEG): container finished" podID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerID="530e418846cf60006385ff34622d6cf71651e9c25e822c681d5acf49c531522a" exitCode=0
Oct 30 03:27:18.898782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.898435    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" event={"ID":"4ab1ff26-0edb-41ef-a952-ae6a17553c4c","Type":"ContainerDied","Data":"530e418846cf60006385ff34622d6cf71651e9c25e822c681d5acf49c531522a"}
Oct 30 03:27:18.898782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.898489    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p" event={"ID":"4ab1ff26-0edb-41ef-a952-ae6a17553c4c","Type":"ContainerDied","Data":"392e83b3f8d73c383a80fd2f8d0df67676bae82219949fb68102b463acf57b47"}
Oct 30 03:27:18.898782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.898511    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="392e83b3f8d73c383a80fd2f8d0df67676bae82219949fb68102b463acf57b47"
Oct 30 03:27:18.898782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.898527    5061 scope.go:122] "RemoveContainer" containerID="a1b9ee68113f7a66bdffb7771cdf89e25c483d879d6da77fcbf3c909ffb3fa62"
Oct 30 03:27:18.912203 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.912185    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:27:18.969998 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.969973    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:18.979744 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.979723    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-token\" (UniqueName: \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-token\") pod \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") "
Oct 30 03:27:18.979873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.979765    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-podinfo\") pod \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") "
Oct 30 03:27:18.979873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.979818    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-socket\") pod \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") "
Oct 30 03:27:18.980012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.979883    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-kube-api-access-m75sr\" (UniqueName: \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-kube-api-access-m75sr\") pod \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") "
Oct 30 03:27:18.980012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.979902    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-certs\") pod \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") "
Oct 30 03:27:18.980012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.979945    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-data\") pod \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") "
Oct 30 03:27:18.980012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.979988    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-envoy\") pod \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") "
Oct 30 03:27:18.980244 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.980067    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istiod-ca-cert\") pod \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") "
Oct 30 03:27:18.980244 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.980104    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-credential-socket\") pod \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\" (UID: \"4ab1ff26-0edb-41ef-a952-ae6a17553c4c\") "
Oct 30 03:27:18.980357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.980290    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-data" pod "4ab1ff26-0edb-41ef-a952-ae6a17553c4c" (UID: "4ab1ff26-0edb-41ef-a952-ae6a17553c4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:18.980760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.980514    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-certs" pod "4ab1ff26-0edb-41ef-a952-ae6a17553c4c" (UID: "4ab1ff26-0edb-41ef-a952-ae6a17553c4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:18.980760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.980585    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istiod-ca-cert" pod "4ab1ff26-0edb-41ef-a952-ae6a17553c4c" (UID: "4ab1ff26-0edb-41ef-a952-ae6a17553c4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:18.980760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.980719    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-credential-socket" pod "4ab1ff26-0edb-41ef-a952-ae6a17553c4c" (UID: "4ab1ff26-0edb-41ef-a952-ae6a17553c4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:18.980995 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.980806    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-socket" pod "4ab1ff26-0edb-41ef-a952-ae6a17553c4c" (UID: "4ab1ff26-0edb-41ef-a952-ae6a17553c4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:18.981066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.981048    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.981108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.981071    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.981108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.981084    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.981108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.981098    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.981254 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.981109    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.982496 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.982473    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-podinfo" pod "4ab1ff26-0edb-41ef-a952-ae6a17553c4c" (UID: "4ab1ff26-0edb-41ef-a952-ae6a17553c4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:27:18.982598 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.982503    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-kube-api-access-m75sr" pod "4ab1ff26-0edb-41ef-a952-ae6a17553c4c" (UID: "4ab1ff26-0edb-41ef-a952-ae6a17553c4c"). InnerVolumeSpecName "kube-api-access-m75sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:18.982598 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.982528    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-envoy" pod "4ab1ff26-0edb-41ef-a952-ae6a17553c4c" (UID: "4ab1ff26-0edb-41ef-a952-ae6a17553c4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:18.982598 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:18.982565    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-token" pod "4ab1ff26-0edb-41ef-a952-ae6a17553c4c" (UID: "4ab1ff26-0edb-41ef-a952-ae6a17553c4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:19.085080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:19.084076    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:19.085080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:19.084112    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:19.085080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:19.084134    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:19.085080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:19.084149    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m75sr\" (UniqueName: \"kubernetes.io/projected/4ab1ff26-0edb-41ef-a952-ae6a17553c4c-kube-api-access-m75sr\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:19.906305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:19.906276    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"
Oct 30 03:27:19.992382 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:19.992351    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"]
Oct 30 03:27:19.996333 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:19.996305    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ac7936-openshift-default-66c9fd8c5-g2q6p"]
Oct 30 03:27:20.603181 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.603143    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"]
Oct 30 03:27:20.603944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.603921    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d440d2d2-15f6-4053-817f-19622b65948b" containerName="maas-api"
Oct 30 03:27:20.603944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.603942    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d440d2d2-15f6-4053-817f-19622b65948b" containerName="maas-api"
Oct 30 03:27:20.604127 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.603962    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy"
Oct 30 03:27:20.604127 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.603968    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy"
Oct 30 03:27:20.604127 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.604001    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy"
Oct 30 03:27:20.604127 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.604007    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy"
Oct 30 03:27:20.604326 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.604162    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d440d2d2-15f6-4053-817f-19622b65948b" containerName="maas-api"
Oct 30 03:27:20.604326 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.604180    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy"
Oct 30 03:27:20.604326 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.604193    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" containerName="istio-proxy"
Oct 30 03:27:20.604612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.604589    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:20.609279 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.609261    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.627317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.627298    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-874ab5-openshift-default-dockercfg-kmjhl\""
Oct 30 03:27:20.638335 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.638307    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"]
Oct 30 03:27:20.702405 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.702375    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28f99f1-cc49-4c8b-8a34-7440b8068712-istiod-ca-cert\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.702538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.702434    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.702538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.702520    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-credential-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.702618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.702559    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-data\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.702676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.702661    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-envoy\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.702736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.702724    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-certs\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.702776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.702756    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-podinfo\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.702883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.702868    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-token\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.702927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.702893    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckv48\" (UniqueName: \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-kube-api-access-ckv48\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804111 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804080    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-certs\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804261 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804125    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-podinfo\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804261 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804175    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-token\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804261 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804197    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckv48\" (UniqueName: \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-kube-api-access-ckv48\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804261 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804255    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28f99f1-cc49-4c8b-8a34-7440b8068712-istiod-ca-cert\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804476 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804287    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804476 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804315    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-credential-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804476 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804339    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-data\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804476 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804438    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-envoy\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804503    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-certs\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804647    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-credential-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804688    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-socket\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.804920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.804809    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-data\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.805120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.805096    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28f99f1-cc49-4c8b-8a34-7440b8068712-istiod-ca-cert\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.806892 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.806861    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-envoy\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.807122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.807103    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-podinfo\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.822005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.821973    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-token\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.822684 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.822666    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckv48\" (UniqueName: \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-kube-api-access-ckv48\") pod \"e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") " pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:20.847588 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.847563    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab1ff26-0edb-41ef-a952-ae6a17553c4c" path="/var/lib/kubelet/pods/4ab1ff26-0edb-41ef-a952-ae6a17553c4c/volumes"
Oct 30 03:27:20.922935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:20.922871    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:21.237692 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:21.237665    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"]
Oct 30 03:27:21.240730 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:27:21.240698    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28f99f1_cc49_4c8b_8a34_7440b8068712.slice/crio-6d04b5198e5f0279407a56766f2c16eb1ae4a29a49ac6624bd21895f4a22d3d5 WatchSource:0}: Error finding container 6d04b5198e5f0279407a56766f2c16eb1ae4a29a49ac6624bd21895f4a22d3d5: Status 404 returned error can't find the container with id 6d04b5198e5f0279407a56766f2c16eb1ae4a29a49ac6624bd21895f4a22d3d5
Oct 30 03:27:21.243434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:21.243397    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:21.243531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:21.243467    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:21.243531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:21.243494    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:21.919944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:21.919902    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf" event={"ID":"e28f99f1-cc49-4c8b-8a34-7440b8068712","Type":"ContainerStarted","Data":"e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af"}
Oct 30 03:27:21.920121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:21.919965    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf" event={"ID":"e28f99f1-cc49-4c8b-8a34-7440b8068712","Type":"ContainerStarted","Data":"6d04b5198e5f0279407a56766f2c16eb1ae4a29a49ac6624bd21895f4a22d3d5"}
Oct 30 03:27:21.923364 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:21.923343    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:22.098236 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:22.098179    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:22.218363 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:22.218212    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:22.944984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:22.944949    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 03:27:22.945417 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:22.945007    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf" podUID="e28f99f1-cc49-4c8b-8a34-7440b8068712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 03:27:23.005997 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:23.005956    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:23.141873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:23.141055    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:23.903365 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:23.903300    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf" podStartSLOduration=3.90327958 podStartE2EDuration="3.90327958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:21.976385235 +0000 UTC m=+2787.278563600" watchObservedRunningTime="2026-10-30 03:27:23.90327958 +0000 UTC m=+2789.205457947"
Oct 30 03:27:23.906077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:23.906047    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"]
Oct 30 03:27:23.939871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:23.933937    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 03:27:23.939871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:23.933994    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf" podUID="e28f99f1-cc49-4c8b-8a34-7440b8068712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 03:27:24.005616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:24.005580    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:24.147125 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:24.147089    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:24.268741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:24.268659    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"]
Oct 30 03:27:24.306206 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:24.306177    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"]
Oct 30 03:27:24.453785 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:24.453748    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:24.582891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:24.582811    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:24.943922 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:24.941181    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 03:27:24.943922 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:24.941241    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf" podUID="e28f99f1-cc49-4c8b-8a34-7440b8068712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 03:27:25.477409 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:27:25.477370    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:27:25.647875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:25.646336    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:25.929265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:25.929222    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 03:27:25.929548 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:25.929516    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf" podUID="e28f99f1-cc49-4c8b-8a34-7440b8068712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 03:27:26.932881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:26.931382    5061 patch_prober.go:29] interesting pod/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 03:27:26.932881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:26.931439    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf" podUID="e28f99f1-cc49-4c8b-8a34-7440b8068712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 03:27:27.353372 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:27.353331    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:27.464156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:27.464128    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:27.582444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:27.582412    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:27.748893 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:27.748765    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg" podUID="df3304e2-e0ca-482e-8f79-52e0724e4663" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:28.353751 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:28.353718    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:28.354789 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:28.354766    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:28.355926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:28.355900    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:27:29.073171 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.073121    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:29.269101 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.269048    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz" podUID="91673351-2242-4930-be49-6482e35194e8" containerName="payload-processing" containerID="cri-o://a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7" gracePeriod=25
Oct 30 03:27:29.306996 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.306963    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld" podUID="dff57095-f4c4-416b-885e-ede0cd4087f3" containerName="payload-pre-processing" containerID="cri-o://13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680" gracePeriod=25
Oct 30 03:27:29.503839 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.503807    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:27:29.574290 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.574257    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3304e2-e0ca-482e-8f79-52e0724e4663-kube-api-access-8q99v\" (UniqueName: \"kubernetes.io/projected/df3304e2-e0ca-482e-8f79-52e0724e4663-kube-api-access-8q99v\") pod \"df3304e2-e0ca-482e-8f79-52e0724e4663\" (UID: \"df3304e2-e0ca-482e-8f79-52e0724e4663\") "
Oct 30 03:27:29.574449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.574401    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df3304e2-e0ca-482e-8f79-52e0724e4663-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3304e2-e0ca-482e-8f79-52e0724e4663-plugins-config-volume\") pod \"df3304e2-e0ca-482e-8f79-52e0724e4663\" (UID: \"df3304e2-e0ca-482e-8f79-52e0724e4663\") "
Oct 30 03:27:29.574775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.574751    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3304e2-e0ca-482e-8f79-52e0724e4663-plugins-config-volume" pod "df3304e2-e0ca-482e-8f79-52e0724e4663" (UID: "df3304e2-e0ca-482e-8f79-52e0724e4663"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:29.574930 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.574912    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3304e2-e0ca-482e-8f79-52e0724e4663-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:29.576575 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.576543    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3304e2-e0ca-482e-8f79-52e0724e4663-kube-api-access-8q99v" pod "df3304e2-e0ca-482e-8f79-52e0724e4663" (UID: "df3304e2-e0ca-482e-8f79-52e0724e4663"). InnerVolumeSpecName "kube-api-access-8q99v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:29.614468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.614437    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"]
Oct 30 03:27:29.676318 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.676278    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q99v\" (UniqueName: \"kubernetes.io/projected/df3304e2-e0ca-482e-8f79-52e0724e4663-kube-api-access-8q99v\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:29.772928 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.772894    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:29.892569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:29.892491    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:30.004012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:30.003981    5061 generic.go:356] "Generic (PLEG): container finished" podID="df3304e2-e0ca-482e-8f79-52e0724e4663" containerID="f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635" exitCode=0
Oct 30 03:27:30.004175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:30.004027    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"
Oct 30 03:27:30.004175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:30.004063    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg" event={"ID":"df3304e2-e0ca-482e-8f79-52e0724e4663","Type":"ContainerDied","Data":"f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635"}
Oct 30 03:27:30.004175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:30.004116    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg" event={"ID":"df3304e2-e0ca-482e-8f79-52e0724e4663","Type":"ContainerDied","Data":"544d007ffb8df67851f6fc93a5589a6daabc0976be60b9264d8c9cf7c777402f"}
Oct 30 03:27:30.004175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:30.004139    5061 scope.go:122] "RemoveContainer" containerID="f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635"
Oct 30 03:27:30.014855 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:30.014823    5061 scope.go:122] "RemoveContainer" containerID="f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635"
Oct 30 03:27:30.015137 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:27:30.015120    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635\": container with ID starting with f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635 not found: ID does not exist" containerID="f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635"
Oct 30 03:27:30.015201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:30.015143    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635"} err="failed to get container status \"f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635\": rpc error: code = NotFound desc = could not find container \"f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635\": container with ID starting with f39347bf767daf84a68021fa765e19df76cea413d8c3f1530f500c74ec106635 not found: ID does not exist"
Oct 30 03:27:30.052656 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:30.052630    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"]
Oct 30 03:27:30.057754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:30.057731    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ac7936-58cdbd6fbd-plvrg"]
Oct 30 03:27:30.847797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:30.847766    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df3304e2-e0ca-482e-8f79-52e0724e4663" path="/var/lib/kubelet/pods/df3304e2-e0ca-482e-8f79-52e0724e4663/volumes"
Oct 30 03:27:31.005671 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:31.005637    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:32.937430 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:32.937398    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:34.256061 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:34.256021    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz" podUID="91673351-2242-4930-be49-6482e35194e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:34.256061 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:34.256038    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld" podUID="dff57095-f4c4-416b-885e-ede0cd4087f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:34.446231 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:34.446200    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:34.615539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:34.615499    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn" podUID="6d01fd3e-ed78-4ad5-b081-89effbaf2338" containerName="payload-processing" containerID="cri-o://7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba" gracePeriod=25
Oct 30 03:27:36.027375 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:36.027345    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:37.571730 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:37.571691    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:38.951277 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:38.951237    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" containerID="cri-o://188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d" gracePeriod=45
Oct 30 03:27:39.228782 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.228756    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 03:27:39.296044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.296005    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:39.397496 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.397466    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tls-certs\") pod \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") "
Oct 30 03:27:39.397666 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.397541    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-dshm\") pod \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") "
Oct 30 03:27:39.397666 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.397560    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-home\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-home\") pod \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") "
Oct 30 03:27:39.397666 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.397664    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tmp-dir\") pod \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") "
Oct 30 03:27:39.397873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.397707    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-kube-api-access-nk6gl\" (UniqueName: \"kubernetes.io/projected/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-kube-api-access-nk6gl\") pod \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") "
Oct 30 03:27:39.397873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.397743    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-model-cache\") pod \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\" (UID: \"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b\") "
Oct 30 03:27:39.397992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.397952    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-home" pod "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" (UID: "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.398047 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.397991    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tmp-dir" pod "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" (UID: "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.398134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.398108    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-model-cache" pod "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" (UID: "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.398488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.398426    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.398526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.398498    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.398526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.398513    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.399808 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.399789    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-dshm" pod "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" (UID: "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:39.399910 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.399891    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-kube-api-access-nk6gl" pod "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" (UID: "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b"). InnerVolumeSpecName "kube-api-access-nk6gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:39.399970 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.399948    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tls-certs" pod "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" (UID: "ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:39.499373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.499290    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.499373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.499323    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk6gl\" (UniqueName: \"kubernetes.io/projected/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-kube-api-access-nk6gl\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.499373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:39.499335    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:40.071388 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.071352    5061 generic.go:356] "Generic (PLEG): container finished" podID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerID="188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d" exitCode=0
Oct 30 03:27:40.071894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.071397    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"
Oct 30 03:27:40.071894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.071447    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" event={"ID":"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b","Type":"ContainerDied","Data":"188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d"}
Oct 30 03:27:40.071894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.071498    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" event={"ID":"ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b","Type":"ContainerDied","Data":"1e2990bd37be7a0f0d7ed4c02044e7f3bf313d4a6978fee29d251ad4e6cd2e29"}
Oct 30 03:27:40.071894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.071525    5061 scope.go:122] "RemoveContainer" containerID="188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d"
Oct 30 03:27:40.081976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.081958    5061 scope.go:122] "RemoveContainer" containerID="188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d"
Oct 30 03:27:40.082221 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:27:40.082204    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d\": container with ID starting with 188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d not found: ID does not exist" containerID="188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d"
Oct 30 03:27:40.082284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.082229    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d"} err="failed to get container status \"188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d\": rpc error: code = NotFound desc = could not find container \"188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d\": container with ID starting with 188f9e3fea8e4cb7a556bb537a989528806cdddac568f8a4c4dde6c862ed490d not found: ID does not exist"
Oct 30 03:27:40.118995 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.118973    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"]
Oct 30 03:27:40.129023 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.129001    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr"]
Oct 30 03:27:40.190322 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.190288    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d9f3ac/distinct-w3-d9f3ac-kserve-5bdcd8d7b6-24phr" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 03:27:40.848066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.848036    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" path="/var/lib/kubelet/pods/ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b/volumes"
Oct 30 03:27:40.867569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:40.867538    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:42.396747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:42.396704    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:44.135731 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:44.135689    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn" podUID="6d01fd3e-ed78-4ad5-b081-89effbaf2338" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:44.159993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:44.159964    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:44.255993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:44.255950    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld" podUID="dff57095-f4c4-416b-885e-ede0cd4087f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:44.256188 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:44.255950    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz" podUID="91673351-2242-4930-be49-6482e35194e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:45.435751 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.435720    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"]
Oct 30 03:27:45.436267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.435962    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b" podUID="30f6ecbf-c78d-4634-844a-7652c4c75658" containerName="maas-api" containerID="cri-o://dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb" gracePeriod=30
Oct 30 03:27:45.519897 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.519832    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:45.723906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.723877    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:27:45.791898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.791871    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-maas-api-tls\") pod \"30f6ecbf-c78d-4634-844a-7652c4c75658\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") "
Oct 30 03:27:45.792096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.792076    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-metrics-certs\") pod \"30f6ecbf-c78d-4634-844a-7652c4c75658\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") "
Oct 30 03:27:45.792154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.792114    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f6ecbf-c78d-4634-844a-7652c4c75658-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f6ecbf-c78d-4634-844a-7652c4c75658-openshift-service-ca\") pod \"30f6ecbf-c78d-4634-844a-7652c4c75658\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") "
Oct 30 03:27:45.792212 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.792150    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f6ecbf-c78d-4634-844a-7652c4c75658-kube-api-access-5wk55\" (UniqueName: \"kubernetes.io/projected/30f6ecbf-c78d-4634-844a-7652c4c75658-kube-api-access-5wk55\") pod \"30f6ecbf-c78d-4634-844a-7652c4c75658\" (UID: \"30f6ecbf-c78d-4634-844a-7652c4c75658\") "
Oct 30 03:27:45.792537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.792502    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f6ecbf-c78d-4634-844a-7652c4c75658-openshift-service-ca" pod "30f6ecbf-c78d-4634-844a-7652c4c75658" (UID: "30f6ecbf-c78d-4634-844a-7652c4c75658"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:45.792943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.792907    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f6ecbf-c78d-4634-844a-7652c4c75658-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:45.794317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.794276    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-maas-api-tls" pod "30f6ecbf-c78d-4634-844a-7652c4c75658" (UID: "30f6ecbf-c78d-4634-844a-7652c4c75658"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:45.794317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.794299    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-metrics-certs" pod "30f6ecbf-c78d-4634-844a-7652c4c75658" (UID: "30f6ecbf-c78d-4634-844a-7652c4c75658"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:45.794465 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.794352    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f6ecbf-c78d-4634-844a-7652c4c75658-kube-api-access-5wk55" pod "30f6ecbf-c78d-4634-844a-7652c4c75658" (UID: "30f6ecbf-c78d-4634-844a-7652c4c75658"). InnerVolumeSpecName "kube-api-access-5wk55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:45.900673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.895594    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:45.900673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.898170    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wk55\" (UniqueName: \"kubernetes.io/projected/30f6ecbf-c78d-4634-844a-7652c4c75658-kube-api-access-5wk55\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:45.900673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.898213    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30f6ecbf-c78d-4634-844a-7652c4c75658-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:45.991138 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:45.991047    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:46.111922 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.111888    5061 generic.go:356] "Generic (PLEG): container finished" podID="30f6ecbf-c78d-4634-844a-7652c4c75658" containerID="dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb" exitCode=0
Oct 30 03:27:46.112095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.111985    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"
Oct 30 03:27:46.112095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.112017    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b" event={"ID":"30f6ecbf-c78d-4634-844a-7652c4c75658","Type":"ContainerDied","Data":"dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb"}
Oct 30 03:27:46.112095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.112073    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b" event={"ID":"30f6ecbf-c78d-4634-844a-7652c4c75658","Type":"ContainerDied","Data":"098754d3e8a6638bcdd1e6a2de1283850b2d69ace9cf1b8fec3974c24dd92c97"}
Oct 30 03:27:46.112215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.112101    5061 scope.go:122] "RemoveContainer" containerID="dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb"
Oct 30 03:27:46.119444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.119418    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:46.123991 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.123973    5061 scope.go:122] "RemoveContainer" containerID="dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb"
Oct 30 03:27:46.124267 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:27:46.124246    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb\": container with ID starting with dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb not found: ID does not exist" containerID="dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb"
Oct 30 03:27:46.124314 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.124275    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb"} err="failed to get container status \"dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb\": rpc error: code = NotFound desc = could not find container \"dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb\": container with ID starting with dcba102f3b987f836b64b76bc4d35d1213411cfc80747b08d1fe33d3cc4b6ddb not found: ID does not exist"
Oct 30 03:27:46.160278 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.160228    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"]
Oct 30 03:27:46.163669 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.163635    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9d3552-799c69bc87-kkr5b"]
Oct 30 03:27:46.234434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.234398    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:46.629776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.629738    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:46.847749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:46.847716    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f6ecbf-c78d-4634-844a-7652c4c75658" path="/var/lib/kubelet/pods/30f6ecbf-c78d-4634-844a-7652c4c75658/volumes"
Oct 30 03:27:47.869503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:47.869465    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:49.596315 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.596287    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:27:49.623863 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.623825    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:27:49.653554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.653513    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91673351-2242-4930-be49-6482e35194e8-kube-api-access-dtbz5\" (UniqueName: \"kubernetes.io/projected/91673351-2242-4930-be49-6482e35194e8-kube-api-access-dtbz5\") pod \"91673351-2242-4930-be49-6482e35194e8\" (UID: \"91673351-2242-4930-be49-6482e35194e8\") "
Oct 30 03:27:49.653718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.653627    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91673351-2242-4930-be49-6482e35194e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91673351-2242-4930-be49-6482e35194e8-plugins-config-volume\") pod \"91673351-2242-4930-be49-6482e35194e8\" (UID: \"91673351-2242-4930-be49-6482e35194e8\") "
Oct 30 03:27:49.654046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.654014    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91673351-2242-4930-be49-6482e35194e8-plugins-config-volume" pod "91673351-2242-4930-be49-6482e35194e8" (UID: "91673351-2242-4930-be49-6482e35194e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:49.654382 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.654359    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91673351-2242-4930-be49-6482e35194e8-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:49.655769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.655742    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91673351-2242-4930-be49-6482e35194e8-kube-api-access-dtbz5" pod "91673351-2242-4930-be49-6482e35194e8" (UID: "91673351-2242-4930-be49-6482e35194e8"). InnerVolumeSpecName "kube-api-access-dtbz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:49.754223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.754191    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:49.755290 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.755267    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff57095-f4c4-416b-885e-ede0cd4087f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff57095-f4c4-416b-885e-ede0cd4087f3-plugins-config-volume\") pod \"dff57095-f4c4-416b-885e-ede0cd4087f3\" (UID: \"dff57095-f4c4-416b-885e-ede0cd4087f3\") "
Oct 30 03:27:49.755456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.755442    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff57095-f4c4-416b-885e-ede0cd4087f3-kube-api-access-tmvn7\" (UniqueName: \"kubernetes.io/projected/dff57095-f4c4-416b-885e-ede0cd4087f3-kube-api-access-tmvn7\") pod \"dff57095-f4c4-416b-885e-ede0cd4087f3\" (UID: \"dff57095-f4c4-416b-885e-ede0cd4087f3\") "
Oct 30 03:27:49.755631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.755614    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff57095-f4c4-416b-885e-ede0cd4087f3-plugins-config-volume" pod "dff57095-f4c4-416b-885e-ede0cd4087f3" (UID: "dff57095-f4c4-416b-885e-ede0cd4087f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:49.755792 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.755780    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff57095-f4c4-416b-885e-ede0cd4087f3-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:49.755837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.755795    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtbz5\" (UniqueName: \"kubernetes.io/projected/91673351-2242-4930-be49-6482e35194e8-kube-api-access-dtbz5\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:49.757553 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.757532    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff57095-f4c4-416b-885e-ede0cd4087f3-kube-api-access-tmvn7" pod "dff57095-f4c4-416b-885e-ede0cd4087f3" (UID: "dff57095-f4c4-416b-885e-ede0cd4087f3"). InnerVolumeSpecName "kube-api-access-tmvn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:49.856941 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:49.856837    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmvn7\" (UniqueName: \"kubernetes.io/projected/dff57095-f4c4-416b-885e-ede0cd4087f3-kube-api-access-tmvn7\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:50.142268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.142183    5061 generic.go:356] "Generic (PLEG): container finished" podID="dff57095-f4c4-416b-885e-ede0cd4087f3" containerID="13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680" exitCode=0
Oct 30 03:27:50.142268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.142225    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"
Oct 30 03:27:50.142502 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.142268    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld" event={"ID":"dff57095-f4c4-416b-885e-ede0cd4087f3","Type":"ContainerDied","Data":"13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680"}
Oct 30 03:27:50.142502 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.142317    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld" event={"ID":"dff57095-f4c4-416b-885e-ede0cd4087f3","Type":"ContainerDied","Data":"bccd3e90ea761fb0c2c17d9781518d84d5e4e0df8e9adb38114a6b5b36b8582c"}
Oct 30 03:27:50.142502 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.142340    5061 scope.go:122] "RemoveContainer" containerID="13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680"
Oct 30 03:27:50.143887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.143864    5061 generic.go:356] "Generic (PLEG): container finished" podID="91673351-2242-4930-be49-6482e35194e8" containerID="a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7" exitCode=0
Oct 30 03:27:50.144017 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.143894    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz" event={"ID":"91673351-2242-4930-be49-6482e35194e8","Type":"ContainerDied","Data":"a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7"}
Oct 30 03:27:50.144017 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.143938    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz" event={"ID":"91673351-2242-4930-be49-6482e35194e8","Type":"ContainerDied","Data":"77d3707dcce54c4f18df66c88c3588a2ad9ecfa207f7191ad02cd95ff3c3e508"}
Oct 30 03:27:50.144017 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.143985    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"
Oct 30 03:27:50.153513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.153411    5061 scope.go:122] "RemoveContainer" containerID="13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680"
Oct 30 03:27:50.153757 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:27:50.153730    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680\": container with ID starting with 13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680 not found: ID does not exist" containerID="13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680"
Oct 30 03:27:50.153876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.153771    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680"} err="failed to get container status \"13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680\": rpc error: code = NotFound desc = could not find container \"13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680\": container with ID starting with 13207558b198cd0f283044e2100d960bb5beca313adfdf20c9e76a759f7da680 not found: ID does not exist"
Oct 30 03:27:50.153876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.153793    5061 scope.go:122] "RemoveContainer" containerID="a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7"
Oct 30 03:27:50.163194 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.163171    5061 scope.go:122] "RemoveContainer" containerID="a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7"
Oct 30 03:27:50.163464 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:27:50.163442    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7\": container with ID starting with a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7 not found: ID does not exist" containerID="a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7"
Oct 30 03:27:50.163527 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.163473    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7"} err="failed to get container status \"a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7\": rpc error: code = NotFound desc = could not find container \"a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7\": container with ID starting with a38ed6b4c0abbfa105a4e60855d88283c9bb89845343b84ef162ff3ba0ef73a7 not found: ID does not exist"
Oct 30 03:27:50.188198 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.188176    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"]
Oct 30 03:27:50.198440 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.198417    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d3552-656ddb4d4c-pgmld"]
Oct 30 03:27:50.228515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.228490    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"]
Oct 30 03:27:50.232223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.232199    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d3552-56c5d44b9b-5nxcz"]
Oct 30 03:27:50.407990 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.407906    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:50.645646 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.645612    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"]
Oct 30 03:27:50.779268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.779187    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:50.847656 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.847624    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91673351-2242-4930-be49-6482e35194e8" path="/var/lib/kubelet/pods/91673351-2242-4930-be49-6482e35194e8/volumes"
Oct 30 03:27:50.848076 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.848062    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff57095-f4c4-416b-885e-ede0cd4087f3" path="/var/lib/kubelet/pods/dff57095-f4c4-416b-885e-ede0cd4087f3/volumes"
Oct 30 03:27:50.899337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:50.899304    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:51.010194 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:51.010156    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:51.401408 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:51.401363    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:51.698321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:51.698238    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"]
Oct 30 03:27:51.750548 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:51.750522    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:53.147429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:53.147397    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:53.288952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:53.288916    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:53.517671 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:53.517584    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:54.135551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:54.135508    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn" podUID="6d01fd3e-ed78-4ad5-b081-89effbaf2338" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:27:54.203589 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:54.203524    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:54.894144 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:54.894113    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 03:27:55.033555 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.033517    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d01fd3e-ed78-4ad5-b081-89effbaf2338-kube-api-access-x77sq\" (UniqueName: \"kubernetes.io/projected/6d01fd3e-ed78-4ad5-b081-89effbaf2338-kube-api-access-x77sq\") pod \"6d01fd3e-ed78-4ad5-b081-89effbaf2338\" (UID: \"6d01fd3e-ed78-4ad5-b081-89effbaf2338\") "
Oct 30 03:27:55.033744 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.033595    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d01fd3e-ed78-4ad5-b081-89effbaf2338-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d01fd3e-ed78-4ad5-b081-89effbaf2338-plugins-config-volume\") pod \"6d01fd3e-ed78-4ad5-b081-89effbaf2338\" (UID: \"6d01fd3e-ed78-4ad5-b081-89effbaf2338\") "
Oct 30 03:27:55.034009 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.033983    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d01fd3e-ed78-4ad5-b081-89effbaf2338-plugins-config-volume" pod "6d01fd3e-ed78-4ad5-b081-89effbaf2338" (UID: "6d01fd3e-ed78-4ad5-b081-89effbaf2338"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:55.035635 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.035609    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d01fd3e-ed78-4ad5-b081-89effbaf2338-kube-api-access-x77sq" pod "6d01fd3e-ed78-4ad5-b081-89effbaf2338" (UID: "6d01fd3e-ed78-4ad5-b081-89effbaf2338"). InnerVolumeSpecName "kube-api-access-x77sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:55.135403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.135337    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d01fd3e-ed78-4ad5-b081-89effbaf2338-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:55.135403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.135364    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x77sq\" (UniqueName: \"kubernetes.io/projected/6d01fd3e-ed78-4ad5-b081-89effbaf2338-kube-api-access-x77sq\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:55.181989 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.181958    5061 generic.go:356] "Generic (PLEG): container finished" podID="6d01fd3e-ed78-4ad5-b081-89effbaf2338" containerID="7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba" exitCode=0
Oct 30 03:27:55.182123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.182015    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"
Oct 30 03:27:55.182123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.182040    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn" event={"ID":"6d01fd3e-ed78-4ad5-b081-89effbaf2338","Type":"ContainerDied","Data":"7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba"}
Oct 30 03:27:55.182123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.182109    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn" event={"ID":"6d01fd3e-ed78-4ad5-b081-89effbaf2338","Type":"ContainerDied","Data":"d6276b8aba1a98a6e1bcb6fc674f8454268c8aa553eeeed151b8d5fd010f462f"}
Oct 30 03:27:55.182251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.182134    5061 scope.go:122] "RemoveContainer" containerID="7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba"
Oct 30 03:27:55.192412 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.192393    5061 scope.go:122] "RemoveContainer" containerID="7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba"
Oct 30 03:27:55.192690 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:27:55.192669    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba\": container with ID starting with 7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba not found: ID does not exist" containerID="7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba"
Oct 30 03:27:55.192784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.192699    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba"} err="failed to get container status \"7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba\": rpc error: code = NotFound desc = could not find container \"7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba\": container with ID starting with 7136d3ccae63d16858aecf003a5d08b20bd82cb61e1af99bb8cee7b7f5ea0eba not found: ID does not exist"
Oct 30 03:27:55.227446 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.227425    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"]
Oct 30 03:27:55.235332 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.235313    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d9f3ac-745464ddc6-d5jzn"]
Oct 30 03:27:55.447739 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:27:55.447649    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:27:55.527008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.526975    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:55.646583 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.646549    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5" podUID="00539a57-b3bc-412c-b30c-166adc0a3844" containerName="payload-pre-processing" containerID="cri-o://06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895" gracePeriod=25
Oct 30 03:27:55.922137 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:55.922113    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:27:56.045505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.045482    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00539a57-b3bc-412c-b30c-166adc0a3844-kube-api-access-lmxjp\" (UniqueName: \"kubernetes.io/projected/00539a57-b3bc-412c-b30c-166adc0a3844-kube-api-access-lmxjp\") pod \"00539a57-b3bc-412c-b30c-166adc0a3844\" (UID: \"00539a57-b3bc-412c-b30c-166adc0a3844\") "
Oct 30 03:27:56.045668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.045534    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00539a57-b3bc-412c-b30c-166adc0a3844-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00539a57-b3bc-412c-b30c-166adc0a3844-plugins-config-volume\") pod \"00539a57-b3bc-412c-b30c-166adc0a3844\" (UID: \"00539a57-b3bc-412c-b30c-166adc0a3844\") "
Oct 30 03:27:56.045998 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.045966    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00539a57-b3bc-412c-b30c-166adc0a3844-plugins-config-volume" pod "00539a57-b3bc-412c-b30c-166adc0a3844" (UID: "00539a57-b3bc-412c-b30c-166adc0a3844"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:56.047760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.047741    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00539a57-b3bc-412c-b30c-166adc0a3844-kube-api-access-lmxjp" pod "00539a57-b3bc-412c-b30c-166adc0a3844" (UID: "00539a57-b3bc-412c-b30c-166adc0a3844"). InnerVolumeSpecName "kube-api-access-lmxjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:56.147693 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.147665    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmxjp\" (UniqueName: \"kubernetes.io/projected/00539a57-b3bc-412c-b30c-166adc0a3844-kube-api-access-lmxjp\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:56.147693 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.147691    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00539a57-b3bc-412c-b30c-166adc0a3844-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:27:56.190162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.190132    5061 generic.go:356] "Generic (PLEG): container finished" podID="00539a57-b3bc-412c-b30c-166adc0a3844" containerID="06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895" exitCode=0
Oct 30 03:27:56.190321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.190176    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"
Oct 30 03:27:56.190321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.190217    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5" event={"ID":"00539a57-b3bc-412c-b30c-166adc0a3844","Type":"ContainerDied","Data":"06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895"}
Oct 30 03:27:56.190321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.190275    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5" event={"ID":"00539a57-b3bc-412c-b30c-166adc0a3844","Type":"ContainerDied","Data":"178d019e120882131337fb76e0892330d4411f82517cf0a9bf3dcf626bebd89e"}
Oct 30 03:27:56.190321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.190302    5061 scope.go:122] "RemoveContainer" containerID="06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895"
Oct 30 03:27:56.200707 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.200688    5061 scope.go:122] "RemoveContainer" containerID="06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895"
Oct 30 03:27:56.200991 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:27:56.200972    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895\": container with ID starting with 06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895 not found: ID does not exist" containerID="06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895"
Oct 30 03:27:56.201050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.200999    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895"} err="failed to get container status \"06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895\": rpc error: code = NotFound desc = could not find container \"06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895\": container with ID starting with 06492c5d06adbac5c24123c72d54fcf034661c930cefc7a091f8e3365c566895 not found: ID does not exist"
Oct 30 03:27:56.235822 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.235801    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"]
Oct 30 03:27:56.239532 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.239513    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-d488dc8bc-86gn5"]
Oct 30 03:27:56.693056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.693022    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"]
Oct 30 03:27:56.726833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.726803    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"]
Oct 30 03:27:56.850261 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.850228    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00539a57-b3bc-412c-b30c-166adc0a3844" path="/var/lib/kubelet/pods/00539a57-b3bc-412c-b30c-166adc0a3844/volumes"
Oct 30 03:27:56.850676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.850662    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d01fd3e-ed78-4ad5-b081-89effbaf2338" path="/var/lib/kubelet/pods/6d01fd3e-ed78-4ad5-b081-89effbaf2338/volumes"
Oct 30 03:27:56.863445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.863422    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:56.987725 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:56.987627    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:57.107815 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:57.107780    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:57.282414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:57.282334    5061 scope.go:122] "RemoveContainer" containerID="d4bfa632ea42f39db38f9b9d6fb18e66020c534fc81c2abf9754730bbdb066cb"
Oct 30 03:27:57.296551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:57.296528    5061 scope.go:122] "RemoveContainer" containerID="530e418846cf60006385ff34622d6cf71651e9c25e822c681d5acf49c531522a"
Oct 30 03:27:57.303651 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:57.303627    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"]
Oct 30 03:27:57.375721 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:57.375693    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:57.956651 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:57.956610    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:59.071271 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:59.071235    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"]
Oct 30 03:27:59.201798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:59.201763    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:59.894037 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:27:59.894004    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:01.201967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.201935    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"]
Oct 30 03:28:01.202950 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.202901    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45" podUID="8aa34fac-44e6-44c0-9666-f0b9d5974a76" containerName="maas-api" containerID="cri-o://3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b" gracePeriod=30
Oct 30 03:28:01.301432 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.301402    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:01.528111 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.528081    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 03:28:01.634585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.634546    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa34fac-44e6-44c0-9666-f0b9d5974a76-kube-api-access-xhbsp\" (UniqueName: \"kubernetes.io/projected/8aa34fac-44e6-44c0-9666-f0b9d5974a76-kube-api-access-xhbsp\") pod \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") "
Oct 30 03:28:01.634912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.634888    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-metrics-certs\") pod \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") "
Oct 30 03:28:01.635097 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.635079    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa34fac-44e6-44c0-9666-f0b9d5974a76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa34fac-44e6-44c0-9666-f0b9d5974a76-openshift-service-ca\") pod \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") "
Oct 30 03:28:01.635248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.635234    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-maas-api-tls\") pod \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\" (UID: \"8aa34fac-44e6-44c0-9666-f0b9d5974a76\") "
Oct 30 03:28:01.639385 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.638684    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa34fac-44e6-44c0-9666-f0b9d5974a76-openshift-service-ca" pod "8aa34fac-44e6-44c0-9666-f0b9d5974a76" (UID: "8aa34fac-44e6-44c0-9666-f0b9d5974a76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:01.642288 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.641001    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-maas-api-tls" pod "8aa34fac-44e6-44c0-9666-f0b9d5974a76" (UID: "8aa34fac-44e6-44c0-9666-f0b9d5974a76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:01.656646 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.656616    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-metrics-certs" pod "8aa34fac-44e6-44c0-9666-f0b9d5974a76" (UID: "8aa34fac-44e6-44c0-9666-f0b9d5974a76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:01.674286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.674030    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa34fac-44e6-44c0-9666-f0b9d5974a76-kube-api-access-xhbsp" pod "8aa34fac-44e6-44c0-9666-f0b9d5974a76" (UID: "8aa34fac-44e6-44c0-9666-f0b9d5974a76"). InnerVolumeSpecName "kube-api-access-xhbsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:01.694287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.694072    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6" podUID="650d875b-fa01-4dee-89a5-f83cabfd99a2" containerName="payload-processing" containerID="cri-o://d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24" gracePeriod=25
Oct 30 03:28:01.730912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.730277    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7" podUID="a036f32a-965c-435a-ade5-8669e02fcb25" containerName="payload-pre-processing" containerID="cri-o://35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20" gracePeriod=25
Oct 30 03:28:01.738690 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.737678    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:01.738690 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.737707    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa34fac-44e6-44c0-9666-f0b9d5974a76-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:01.738690 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.737722    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa34fac-44e6-44c0-9666-f0b9d5974a76-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:01.738690 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.737737    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhbsp\" (UniqueName: \"kubernetes.io/projected/8aa34fac-44e6-44c0-9666-f0b9d5974a76-kube-api-access-xhbsp\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:01.779477 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.778805    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:01.902925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:01.902821    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:02.014025 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.013997    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:28:02.018170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.018138    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:02.042349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.042329    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/650d875b-fa01-4dee-89a5-f83cabfd99a2-kube-api-access-kmznj\" (UniqueName: \"kubernetes.io/projected/650d875b-fa01-4dee-89a5-f83cabfd99a2-kube-api-access-kmznj\") pod \"650d875b-fa01-4dee-89a5-f83cabfd99a2\" (UID: \"650d875b-fa01-4dee-89a5-f83cabfd99a2\") "
Oct 30 03:28:02.042648 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.042625    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/650d875b-fa01-4dee-89a5-f83cabfd99a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650d875b-fa01-4dee-89a5-f83cabfd99a2-plugins-config-volume\") pod \"650d875b-fa01-4dee-89a5-f83cabfd99a2\" (UID: \"650d875b-fa01-4dee-89a5-f83cabfd99a2\") "
Oct 30 03:28:02.043034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.042999    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/650d875b-fa01-4dee-89a5-f83cabfd99a2-plugins-config-volume" pod "650d875b-fa01-4dee-89a5-f83cabfd99a2" (UID: "650d875b-fa01-4dee-89a5-f83cabfd99a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:02.044558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.044535    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650d875b-fa01-4dee-89a5-f83cabfd99a2-kube-api-access-kmznj" pod "650d875b-fa01-4dee-89a5-f83cabfd99a2" (UID: "650d875b-fa01-4dee-89a5-f83cabfd99a2"). InnerVolumeSpecName "kube-api-access-kmznj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:02.046443 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.046423    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:28:02.144091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.144061    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a036f32a-965c-435a-ade5-8669e02fcb25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a036f32a-965c-435a-ade5-8669e02fcb25-plugins-config-volume\") pod \"a036f32a-965c-435a-ade5-8669e02fcb25\" (UID: \"a036f32a-965c-435a-ade5-8669e02fcb25\") "
Oct 30 03:28:02.144254 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.144152    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a036f32a-965c-435a-ade5-8669e02fcb25-kube-api-access-jh697\" (UniqueName: \"kubernetes.io/projected/a036f32a-965c-435a-ade5-8669e02fcb25-kube-api-access-jh697\") pod \"a036f32a-965c-435a-ade5-8669e02fcb25\" (UID: \"a036f32a-965c-435a-ade5-8669e02fcb25\") "
Oct 30 03:28:02.144434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.144411    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a036f32a-965c-435a-ade5-8669e02fcb25-plugins-config-volume" pod "a036f32a-965c-435a-ade5-8669e02fcb25" (UID: "a036f32a-965c-435a-ade5-8669e02fcb25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:02.144663 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.144644    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a036f32a-965c-435a-ade5-8669e02fcb25-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:02.144697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.144671    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650d875b-fa01-4dee-89a5-f83cabfd99a2-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:02.144697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.144687    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmznj\" (UniqueName: \"kubernetes.io/projected/650d875b-fa01-4dee-89a5-f83cabfd99a2-kube-api-access-kmznj\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:02.146348 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.146329    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a036f32a-965c-435a-ade5-8669e02fcb25-kube-api-access-jh697" pod "a036f32a-965c-435a-ade5-8669e02fcb25" (UID: "a036f32a-965c-435a-ade5-8669e02fcb25"). InnerVolumeSpecName "kube-api-access-jh697". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:02.241442 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.241412    5061 generic.go:356] "Generic (PLEG): container finished" podID="8aa34fac-44e6-44c0-9666-f0b9d5974a76" containerID="3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b" exitCode=0
Oct 30 03:28:02.241925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.241468    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"
Oct 30 03:28:02.241925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.241503    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45" event={"ID":"8aa34fac-44e6-44c0-9666-f0b9d5974a76","Type":"ContainerDied","Data":"3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b"}
Oct 30 03:28:02.241925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.241595    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45" event={"ID":"8aa34fac-44e6-44c0-9666-f0b9d5974a76","Type":"ContainerDied","Data":"40ea5af3f04dcbe2ae5bab40bb9b8d4d7d5ee609bee74086f050f2d4b5f7c449"}
Oct 30 03:28:02.241925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.241624    5061 scope.go:122] "RemoveContainer" containerID="3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b"
Oct 30 03:28:02.243114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.243092    5061 generic.go:356] "Generic (PLEG): container finished" podID="650d875b-fa01-4dee-89a5-f83cabfd99a2" containerID="d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24" exitCode=0
Oct 30 03:28:02.243229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.243175    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6" event={"ID":"650d875b-fa01-4dee-89a5-f83cabfd99a2","Type":"ContainerDied","Data":"d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24"}
Oct 30 03:28:02.243229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.243195    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"
Oct 30 03:28:02.243341 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.243229    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6" event={"ID":"650d875b-fa01-4dee-89a5-f83cabfd99a2","Type":"ContainerDied","Data":"6e1d7434e7066e5aa3dfdc5452bd8959f9f80f69286c8893e236ca7c5671f408"}
Oct 30 03:28:02.244683 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.244659    5061 generic.go:356] "Generic (PLEG): container finished" podID="a036f32a-965c-435a-ade5-8669e02fcb25" containerID="35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20" exitCode=0
Oct 30 03:28:02.244775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.244707    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7" event={"ID":"a036f32a-965c-435a-ade5-8669e02fcb25","Type":"ContainerDied","Data":"35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20"}
Oct 30 03:28:02.244775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.244754    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7" event={"ID":"a036f32a-965c-435a-ade5-8669e02fcb25","Type":"ContainerDied","Data":"d9c33d7531487a863ce864c84a4ef8996489943d76ec85d99447b11c0cacdfb6"}
Oct 30 03:28:02.244962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.244942    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"
Oct 30 03:28:02.245623 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.245604    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh697\" (UniqueName: \"kubernetes.io/projected/a036f32a-965c-435a-ade5-8669e02fcb25-kube-api-access-jh697\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:02.252010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.251984    5061 scope.go:122] "RemoveContainer" containerID="3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b"
Oct 30 03:28:02.252313 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:02.252281    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b\": container with ID starting with 3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b not found: ID does not exist" containerID="3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b"
Oct 30 03:28:02.252419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.252321    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b"} err="failed to get container status \"3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b\": rpc error: code = NotFound desc = could not find container \"3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b\": container with ID starting with 3d8a01b7d8892987e7fc2fd84e94d2e3d6e529f8ab6dec5118da4b608181442b not found: ID does not exist"
Oct 30 03:28:02.252419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.252345    5061 scope.go:122] "RemoveContainer" containerID="d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24"
Oct 30 03:28:02.262142 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.262123    5061 scope.go:122] "RemoveContainer" containerID="d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24"
Oct 30 03:28:02.262387 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:02.262371    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24\": container with ID starting with d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24 not found: ID does not exist" containerID="d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24"
Oct 30 03:28:02.262447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.262392    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24"} err="failed to get container status \"d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24\": rpc error: code = NotFound desc = could not find container \"d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24\": container with ID starting with d25f1f872cdf839921b9cffd5ea999a3be89a642321d087fe7adbe1fc3f0ba24 not found: ID does not exist"
Oct 30 03:28:02.262447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.262406    5061 scope.go:122] "RemoveContainer" containerID="35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20"
Oct 30 03:28:02.270835 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.270818    5061 scope.go:122] "RemoveContainer" containerID="35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20"
Oct 30 03:28:02.271117 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:02.271097    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20\": container with ID starting with 35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20 not found: ID does not exist" containerID="35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20"
Oct 30 03:28:02.271181 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.271126    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20"} err="failed to get container status \"35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20\": rpc error: code = NotFound desc = could not find container \"35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20\": container with ID starting with 35a0568d10e6c7928075997ca723fa1a888efac3e24e5973bcd00d92dd285f20 not found: ID does not exist"
Oct 30 03:28:02.313401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.313376    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"]
Oct 30 03:28:02.315908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.315887    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-7b9f9d8b8c-xxdr7"]
Oct 30 03:28:02.344675 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.344651    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"]
Oct 30 03:28:02.348587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.348567    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-6dc95c5c5d-6mzq6"]
Oct 30 03:28:02.377028 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.376996    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:02.377805 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.377776    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"]
Oct 30 03:28:02.381472 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.381450    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d9f3ac-7fcdcc4df5-dkl45"]
Oct 30 03:28:02.488704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.488676    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:02.848182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.848147    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="650d875b-fa01-4dee-89a5-f83cabfd99a2" path="/var/lib/kubelet/pods/650d875b-fa01-4dee-89a5-f83cabfd99a2/volumes"
Oct 30 03:28:02.848642 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.848626    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa34fac-44e6-44c0-9666-f0b9d5974a76" path="/var/lib/kubelet/pods/8aa34fac-44e6-44c0-9666-f0b9d5974a76/volumes"
Oct 30 03:28:02.849079 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:02.849060    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a036f32a-965c-435a-ade5-8669e02fcb25" path="/var/lib/kubelet/pods/a036f32a-965c-435a-ade5-8669e02fcb25/volumes"
Oct 30 03:28:03.657410 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:03.657346    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:04.924639 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:04.924607    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:05.042884 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:05.042827    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:05.554464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:05.554407    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:06.720678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:06.720640    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d" podUID="2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" containerName="main" containerID="cri-o://befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd" gracePeriod=45
Oct 30 03:28:06.993648 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:06.993617    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:28:07.104954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.104915    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-dshm\") pod \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") "
Oct 30 03:28:07.105156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.104965    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tls-certs\") pod \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") "
Oct 30 03:28:07.105156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.105018    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tmp-dir\") pod \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") "
Oct 30 03:28:07.105156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.105144    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-home\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-home\") pod \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") "
Oct 30 03:28:07.105317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.105277    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-model-cache\") pod \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") "
Oct 30 03:28:07.105317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.105309    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-kube-api-access-jdp6c\" (UniqueName: \"kubernetes.io/projected/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-kube-api-access-jdp6c\") pod \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\" (UID: \"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7\") "
Oct 30 03:28:07.105487 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.105448    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tmp-dir" pod "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" (UID: "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:07.105487 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.105467    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-home" pod "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" (UID: "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:07.105634 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.105610    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-model-cache" pod "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" (UID: "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:07.105790 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.105769    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:07.105790 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.105789    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:07.106010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.105802    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:07.107381 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.107351    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-dshm" pod "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" (UID: "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:07.107499 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.107476    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-kube-api-access-jdp6c" pod "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" (UID: "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7"). InnerVolumeSpecName "kube-api-access-jdp6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:07.107663 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.107648    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tls-certs" pod "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" (UID: "2cfd344b-a097-4d6e-9cf6-cefad49e3ef7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:07.207150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.207115    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdp6c\" (UniqueName: \"kubernetes.io/projected/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-kube-api-access-jdp6c\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:07.207150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.207145    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:07.207150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.207155    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:07.277585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.277493    5061 generic.go:356] "Generic (PLEG): container finished" podID="2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" containerID="befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd" exitCode=0
Oct 30 03:28:07.277585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.277543    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"
Oct 30 03:28:07.277834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.277585    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d" event={"ID":"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7","Type":"ContainerDied","Data":"befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd"}
Oct 30 03:28:07.277834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.277638    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d" event={"ID":"2cfd344b-a097-4d6e-9cf6-cefad49e3ef7","Type":"ContainerDied","Data":"e18fd30e0deefe9b5cb7896f606bbdfc47831103da9bdcb7b5f3e6731627168b"}
Oct 30 03:28:07.277834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.277659    5061 scope.go:122] "RemoveContainer" containerID="befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd"
Oct 30 03:28:07.288741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.288707    5061 scope.go:122] "RemoveContainer" containerID="befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd"
Oct 30 03:28:07.289184 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:07.289155    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd\": container with ID starting with befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd not found: ID does not exist" containerID="befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd"
Oct 30 03:28:07.289290 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.289196    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd"} err="failed to get container status \"befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd\": rpc error: code = NotFound desc = could not find container \"befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd\": container with ID starting with befd65e91bdb5aab7f16bde70bb73ebfeb6843b298e89ac9b66bcb7c85300cdd not found: ID does not exist"
Oct 30 03:28:07.325046 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.325011    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"]
Oct 30 03:28:07.329972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.329949    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-4b5459/rate-test-model-4b5459-kserve-54c4c77989-9vk6d"]
Oct 30 03:28:07.363468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:07.363435    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:08.122318 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:08.122278    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"]
Oct 30 03:28:08.259075 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:08.259040    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:08.376348 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:08.376260    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:08.848826 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:08.848787    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" path="/var/lib/kubelet/pods/2cfd344b-a097-4d6e-9cf6-cefad49e3ef7/volumes"
Oct 30 03:28:08.920656 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:08.920618    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:09.049675 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:09.049639    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:09.241077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:09.241001    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"]
Oct 30 03:28:09.241512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:09.241250    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" containerID="cri-o://017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c" gracePeriod=30
Oct 30 03:28:09.289716 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:09.289680    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:09.425007 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:09.424971    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:09.563537 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:09.563504    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:09.571682 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:09.571658    5061 patch_prober.go:29] interesting pod/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:28:09.571805 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:09.571735    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:28:10.320660 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:10.320611    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:10.449894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:10.448815    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:11.201895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:11.201824    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:12.327245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.327193    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8" podUID="8c8b1046-7371-4f95-a724-532110f24a62" containerName="main" containerID="cri-o://bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8" gracePeriod=45
Oct 30 03:28:12.431686 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.431644    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8" podUID="8c8b1046-7371-4f95-a724-532110f24a62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.169:8000/ready\": dial tcp 10.132.0.169:8000: connect: connection refused"
Oct 30 03:28:12.587655 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.587597    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:28:12.684368 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.684335    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8b1046-7371-4f95-a724-532110f24a62-kube-api-access-6xs9d\" (UniqueName: \"kubernetes.io/projected/8c8b1046-7371-4f95-a724-532110f24a62-kube-api-access-6xs9d\") pod \"8c8b1046-7371-4f95-a724-532110f24a62\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") "
Oct 30 03:28:12.684539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.684389    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-model-cache\") pod \"8c8b1046-7371-4f95-a724-532110f24a62\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") "
Oct 30 03:28:12.684539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.684425    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-home\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-home\") pod \"8c8b1046-7371-4f95-a724-532110f24a62\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") "
Oct 30 03:28:12.684539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.684475    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-tmp-dir\") pod \"8c8b1046-7371-4f95-a724-532110f24a62\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") "
Oct 30 03:28:12.684676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.684616    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-dshm\") pod \"8c8b1046-7371-4f95-a724-532110f24a62\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") "
Oct 30 03:28:12.684676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.684658    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c8b1046-7371-4f95-a724-532110f24a62-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c8b1046-7371-4f95-a724-532110f24a62-tls-certs\") pod \"8c8b1046-7371-4f95-a724-532110f24a62\" (UID: \"8c8b1046-7371-4f95-a724-532110f24a62\") "
Oct 30 03:28:12.684785 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.684697    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-home" pod "8c8b1046-7371-4f95-a724-532110f24a62" (UID: "8c8b1046-7371-4f95-a724-532110f24a62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:12.684785 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.684713    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-model-cache" pod "8c8b1046-7371-4f95-a724-532110f24a62" (UID: "8c8b1046-7371-4f95-a724-532110f24a62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:12.684785 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.684763    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-tmp-dir" pod "8c8b1046-7371-4f95-a724-532110f24a62" (UID: "8c8b1046-7371-4f95-a724-532110f24a62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:12.685186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.685166    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:12.685259 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.685189    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:12.685259 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.685198    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:12.686759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.686726    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-dshm" pod "8c8b1046-7371-4f95-a724-532110f24a62" (UID: "8c8b1046-7371-4f95-a724-532110f24a62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:12.686899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.686792    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8b1046-7371-4f95-a724-532110f24a62-kube-api-access-6xs9d" pod "8c8b1046-7371-4f95-a724-532110f24a62" (UID: "8c8b1046-7371-4f95-a724-532110f24a62"). InnerVolumeSpecName "kube-api-access-6xs9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:12.686899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.686832    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c8b1046-7371-4f95-a724-532110f24a62-tls-certs" pod "8c8b1046-7371-4f95-a724-532110f24a62" (UID: "8c8b1046-7371-4f95-a724-532110f24a62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:12.785961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.785931    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c8b1046-7371-4f95-a724-532110f24a62-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:12.785961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.785955    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c8b1046-7371-4f95-a724-532110f24a62-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:12.785961 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.785965    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xs9d\" (UniqueName: \"kubernetes.io/projected/8c8b1046-7371-4f95-a724-532110f24a62-kube-api-access-6xs9d\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:12.931282 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:12.931250    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:13.122837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:13.122741    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5" podUID="03805bd7-ff01-4eb0-ae47-91d75b7c3ab6" containerName="payload-pre-processing" containerID="cri-o://dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b" gracePeriod=25
Oct 30 03:28:13.317919 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:13.317887    5061 generic.go:356] "Generic (PLEG): container finished" podID="8c8b1046-7371-4f95-a724-532110f24a62" containerID="bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8" exitCode=0
Oct 30 03:28:13.318096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:13.317935    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"
Oct 30 03:28:13.318096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:13.317976    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8" event={"ID":"8c8b1046-7371-4f95-a724-532110f24a62","Type":"ContainerDied","Data":"bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8"}
Oct 30 03:28:13.318096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:13.318021    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8" event={"ID":"8c8b1046-7371-4f95-a724-532110f24a62","Type":"ContainerDied","Data":"31236a46c0ccbcf8f8e176a285e7dcb974f22ac1c284d56d67b8450f6cc8511c"}
Oct 30 03:28:13.318096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:13.318038    5061 scope.go:122] "RemoveContainer" containerID="bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8"
Oct 30 03:28:13.327474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:13.327452    5061 scope.go:122] "RemoveContainer" containerID="bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8"
Oct 30 03:28:13.327803 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:13.327735    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8\": container with ID starting with bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8 not found: ID does not exist" containerID="bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8"
Oct 30 03:28:13.327803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:13.327768    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8"} err="failed to get container status \"bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8\": rpc error: code = NotFound desc = could not find container \"bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8\": container with ID starting with bd232c55f3790875abffb6f2bb09a6d0f5fc5d8fdf341d4cf927cc094454d2f8 not found: ID does not exist"
Oct 30 03:28:13.364059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:13.364037    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"]
Oct 30 03:28:13.374352 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:13.374297    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-8b0787/rate-test-model-8b0787-kserve-5f54b46bd5-ml2v8"]
Oct 30 03:28:14.041563 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.041530    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:14.092155 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.092114    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main" containerID="cri-o://e0cb695ac54f015afed6b71160ad595937269bfa787166cd0f6679f91f4788d1" gracePeriod=45
Oct 30 03:28:14.192075 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.192040    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:14.384759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.384600    5061 generic.go:356] "Generic (PLEG): container finished" podID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerID="e0cb695ac54f015afed6b71160ad595937269bfa787166cd0f6679f91f4788d1" exitCode=0
Oct 30 03:28:14.384759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.384675    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" event={"ID":"6a59ff6b-e6b4-459a-9b33-26088de3a5c9","Type":"ContainerDied","Data":"e0cb695ac54f015afed6b71160ad595937269bfa787166cd0f6679f91f4788d1"}
Oct 30 03:28:14.419356 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.419322    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"]
Oct 30 03:28:14.420226 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420198    5061 cpu_manager.go:414] "containerMap: removing container" podUID="00539a57-b3bc-412c-b30c-166adc0a3844" containerName="payload-pre-processing"
Oct 30 03:28:14.420323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420229    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00539a57-b3bc-412c-b30c-166adc0a3844" containerName="payload-pre-processing"
Oct 30 03:28:14.420323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420255    5061 cpu_manager.go:414] "containerMap: removing container" podUID="91673351-2242-4930-be49-6482e35194e8" containerName="payload-processing"
Oct 30 03:28:14.420323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420268    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91673351-2242-4930-be49-6482e35194e8" containerName="payload-processing"
Oct 30 03:28:14.420323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420290    5061 cpu_manager.go:414] "containerMap: removing container" podUID="30f6ecbf-c78d-4634-844a-7652c4c75658" containerName="maas-api"
Oct 30 03:28:14.420323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420298    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f6ecbf-c78d-4634-844a-7652c4c75658" containerName="maas-api"
Oct 30 03:28:14.420323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420313    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8aa34fac-44e6-44c0-9666-f0b9d5974a76" containerName="maas-api"
Oct 30 03:28:14.420323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420324    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa34fac-44e6-44c0-9666-f0b9d5974a76" containerName="maas-api"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420341    5061 cpu_manager.go:414] "containerMap: removing container" podUID="a036f32a-965c-435a-ade5-8669e02fcb25" containerName="payload-pre-processing"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420352    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a036f32a-965c-435a-ade5-8669e02fcb25" containerName="payload-pre-processing"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420370    5061 cpu_manager.go:414] "containerMap: removing container" podUID="dff57095-f4c4-416b-885e-ede0cd4087f3" containerName="payload-pre-processing"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420378    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff57095-f4c4-416b-885e-ede0cd4087f3" containerName="payload-pre-processing"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420394    5061 cpu_manager.go:414] "containerMap: removing container" podUID="650d875b-fa01-4dee-89a5-f83cabfd99a2" containerName="payload-processing"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420404    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="650d875b-fa01-4dee-89a5-f83cabfd99a2" containerName="payload-processing"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420422    5061 cpu_manager.go:414] "containerMap: removing container" podUID="6d01fd3e-ed78-4ad5-b081-89effbaf2338" containerName="payload-processing"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420429    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d01fd3e-ed78-4ad5-b081-89effbaf2338" containerName="payload-processing"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420442    5061 cpu_manager.go:414] "containerMap: removing container" podUID="df3304e2-e0ca-482e-8f79-52e0724e4663" containerName="payload-processing"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420449    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df3304e2-e0ca-482e-8f79-52e0724e4663" containerName="payload-processing"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420459    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8c8b1046-7371-4f95-a724-532110f24a62" containerName="main"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420466    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8b1046-7371-4f95-a724-532110f24a62" containerName="main"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420487    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420494    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420523    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" containerName="main"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420530    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" containerName="main"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420687    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="91673351-2242-4930-be49-6482e35194e8" containerName="payload-processing"
Oct 30 03:28:14.420697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420705    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f6ecbf-c78d-4634-844a-7652c4c75658" containerName="maas-api"
Oct 30 03:28:14.421614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420715    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa34fac-44e6-44c0-9666-f0b9d5974a76" containerName="maas-api"
Oct 30 03:28:14.421614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420725    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="a036f32a-965c-435a-ade5-8669e02fcb25" containerName="payload-pre-processing"
Oct 30 03:28:14.421614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420736    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff57095-f4c4-416b-885e-ede0cd4087f3" containerName="payload-pre-processing"
Oct 30 03:28:14.421614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420745    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="650d875b-fa01-4dee-89a5-f83cabfd99a2" containerName="payload-processing"
Oct 30 03:28:14.421614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420754    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d01fd3e-ed78-4ad5-b081-89effbaf2338" containerName="payload-processing"
Oct 30 03:28:14.421614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420762    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="df3304e2-e0ca-482e-8f79-52e0724e4663" containerName="payload-processing"
Oct 30 03:28:14.421614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420772    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8b1046-7371-4f95-a724-532110f24a62" containerName="main"
Oct 30 03:28:14.421614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420783    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6871e6-84d2-42fb-b7b2-060e5f3fcc8b" containerName="main"
Oct 30 03:28:14.421614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420804    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cfd344b-a097-4d6e-9cf6-cefad49e3ef7" containerName="main"
Oct 30 03:28:14.421614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.420817    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="00539a57-b3bc-412c-b30c-166adc0a3844" containerName="payload-pre-processing"
Oct 30 03:28:14.432901 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.425652    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:28:14.438447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.437620    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 03:28:14.450116 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.450036    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-055daa-dockercfg-29l2b\""
Oct 30 03:28:14.450342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.450296    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-055daa\""
Oct 30 03:28:14.453430 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.453402    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"]
Oct 30 03:28:14.505917 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.505880    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-model-cache\") pod \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") "
Oct 30 03:28:14.506080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.505952    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tmp-dir\") pod \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") "
Oct 30 03:28:14.506080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.505990    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-home\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-home\") pod \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") "
Oct 30 03:28:14.506080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.506040    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-dshm\") pod \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") "
Oct 30 03:28:14.506267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.506079    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-kube-api-access-6ccq2\" (UniqueName: \"kubernetes.io/projected/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-kube-api-access-6ccq2\") pod \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") "
Oct 30 03:28:14.506267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.506164    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-model-cache" pod "6a59ff6b-e6b4-459a-9b33-26088de3a5c9" (UID: "6a59ff6b-e6b4-459a-9b33-26088de3a5c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.506267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.506252    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tls-certs\") pod \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\" (UID: \"6a59ff6b-e6b4-459a-9b33-26088de3a5c9\") "
Oct 30 03:28:14.506616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.506566    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-home" pod "6a59ff6b-e6b4-459a-9b33-26088de3a5c9" (UID: "6a59ff6b-e6b4-459a-9b33-26088de3a5c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.506616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.506577    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4d6\" (UniqueName: \"kubernetes.io/projected/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-kube-api-access-hm4d6\") pod \"payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv\" (UID: \"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:28:14.507024 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.507002    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tmp-dir" pod "6a59ff6b-e6b4-459a-9b33-26088de3a5c9" (UID: "6a59ff6b-e6b4-459a-9b33-26088de3a5c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.507340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.507247    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv\" (UID: \"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:28:14.507500 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.507466    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.507500 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.507497    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.507641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.507510    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.510022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.509981    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-dshm" pod "6a59ff6b-e6b4-459a-9b33-26088de3a5c9" (UID: "6a59ff6b-e6b4-459a-9b33-26088de3a5c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.511032 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.511003    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-kube-api-access-6ccq2" pod "6a59ff6b-e6b4-459a-9b33-26088de3a5c9" (UID: "6a59ff6b-e6b4-459a-9b33-26088de3a5c9"). InnerVolumeSpecName "kube-api-access-6ccq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:14.511129 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.511047    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tls-certs" pod "6a59ff6b-e6b4-459a-9b33-26088de3a5c9" (UID: "6a59ff6b-e6b4-459a-9b33-26088de3a5c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:14.583120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.583090    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 03:28:14.607211 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.607182    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"]
Oct 30 03:28:14.607887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.607868    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy"
Oct 30 03:28:14.607970 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.607890    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy"
Oct 30 03:28:14.607970 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.607912    5061 cpu_manager.go:414] "containerMap: removing container" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main"
Oct 30 03:28:14.607970 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.607917    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main"
Oct 30 03:28:14.608072 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.608019    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" containerName="istio-proxy"
Oct 30 03:28:14.608072 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.608029    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" containerName="main"
Oct 30 03:28:14.608309 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.608288    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv\" (UID: \"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:28:14.608497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.608476    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4d6\" (UniqueName: \"kubernetes.io/projected/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-kube-api-access-hm4d6\") pod \"payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv\" (UID: \"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:28:14.608630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.608615    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.608671 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.608637    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.608671 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.608650    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ccq2\" (UniqueName: \"kubernetes.io/projected/6a59ff6b-e6b4-459a-9b33-26088de3a5c9-kube-api-access-6ccq2\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.609105 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.609086    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv\" (UID: \"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:28:14.611515 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.611495    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:28:14.629147 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.629117    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4d6\" (UniqueName: \"kubernetes.io/projected/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-kube-api-access-hm4d6\") pod \"payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv\" (UID: \"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:28:14.630509 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.630476    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"]
Oct 30 03:28:14.709747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.709525    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3aae8e9-b846-4409-877b-618c0b69302b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3aae8e9-b846-4409-877b-618c0b69302b-istio-podinfo\") pod \"f3aae8e9-b846-4409-877b-618c0b69302b\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") "
Oct 30 03:28:14.709747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.709650    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-istio-token\" (UniqueName: \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-istio-token\") pod \"f3aae8e9-b846-4409-877b-618c0b69302b\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") "
Oct 30 03:28:14.709747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.709717    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-kube-api-access-zccgz\" (UniqueName: \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-kube-api-access-zccgz\") pod \"f3aae8e9-b846-4409-877b-618c0b69302b\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") "
Oct 30 03:28:14.710382 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.709910    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-certs\") pod \"f3aae8e9-b846-4409-877b-618c0b69302b\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") "
Oct 30 03:28:14.710382 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.710065    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-data\") pod \"f3aae8e9-b846-4409-877b-618c0b69302b\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") "
Oct 30 03:28:14.710382 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.710226    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-envoy\") pod \"f3aae8e9-b846-4409-877b-618c0b69302b\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") "
Oct 30 03:28:14.710382 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.710348    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-certs" pod "f3aae8e9-b846-4409-877b-618c0b69302b" (UID: "f3aae8e9-b846-4409-877b-618c0b69302b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.710644 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.710404    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-data" pod "f3aae8e9-b846-4409-877b-618c0b69302b" (UID: "f3aae8e9-b846-4409-877b-618c0b69302b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.711512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.711020    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-credential-socket\") pod \"f3aae8e9-b846-4409-877b-618c0b69302b\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") "
Oct 30 03:28:14.711786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.711457    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-credential-socket" pod "f3aae8e9-b846-4409-877b-618c0b69302b" (UID: "f3aae8e9-b846-4409-877b-618c0b69302b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.711786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.711620    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3aae8e9-b846-4409-877b-618c0b69302b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3aae8e9-b846-4409-877b-618c0b69302b-istiod-ca-cert\") pod \"f3aae8e9-b846-4409-877b-618c0b69302b\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") "
Oct 30 03:28:14.712252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.712212    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3aae8e9-b846-4409-877b-618c0b69302b-istiod-ca-cert" pod "f3aae8e9-b846-4409-877b-618c0b69302b" (UID: "f3aae8e9-b846-4409-877b-618c0b69302b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:14.712343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.712314    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-socket\") pod \"f3aae8e9-b846-4409-877b-618c0b69302b\" (UID: \"f3aae8e9-b846-4409-877b-618c0b69302b\") "
Oct 30 03:28:14.713366 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.713319    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-socket" pod "f3aae8e9-b846-4409-877b-618c0b69302b" (UID: "f3aae8e9-b846-4409-877b-618c0b69302b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.713521 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.713194    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j\" (UID: \"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:28:14.713998 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.713969    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6bbm\" (UniqueName: \"kubernetes.io/projected/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-kube-api-access-v6bbm\") pod \"payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j\" (UID: \"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:28:14.714456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.714405    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-envoy" pod "f3aae8e9-b846-4409-877b-618c0b69302b" (UID: "f3aae8e9-b846-4409-877b-618c0b69302b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:14.714456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.714441    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3aae8e9-b846-4409-877b-618c0b69302b-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.714633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.714471    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.714633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.714471    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-kube-api-access-zccgz" pod "f3aae8e9-b846-4409-877b-618c0b69302b" (UID: "f3aae8e9-b846-4409-877b-618c0b69302b"). InnerVolumeSpecName "kube-api-access-zccgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:14.714633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.714497    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.714633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.714515    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.714633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.714513    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-istio-token" pod "f3aae8e9-b846-4409-877b-618c0b69302b" (UID: "f3aae8e9-b846-4409-877b-618c0b69302b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:14.714633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.714530    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.714633 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.714555    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3aae8e9-b846-4409-877b-618c0b69302b-istio-podinfo" pod "f3aae8e9-b846-4409-877b-618c0b69302b" (UID: "f3aae8e9-b846-4409-877b-618c0b69302b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:28:14.740729 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.740657    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:28:14.816807 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.816716    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6bbm\" (UniqueName: \"kubernetes.io/projected/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-kube-api-access-v6bbm\") pod \"payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j\" (UID: \"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:28:14.817281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.817072    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j\" (UID: \"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:28:14.819973 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.818675    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j\" (UID: \"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:28:14.819973 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.819503    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3aae8e9-b846-4409-877b-618c0b69302b-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.819973 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.819536    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3aae8e9-b846-4409-877b-618c0b69302b-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.819973 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.819551    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.819973 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.819571    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zccgz\" (UniqueName: \"kubernetes.io/projected/f3aae8e9-b846-4409-877b-618c0b69302b-kube-api-access-zccgz\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:14.852956 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.846078    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6bbm\" (UniqueName: \"kubernetes.io/projected/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-kube-api-access-v6bbm\") pod \"payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j\" (UID: \"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:28:14.887148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.886923    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8b1046-7371-4f95-a724-532110f24a62" path="/var/lib/kubelet/pods/8c8b1046-7371-4f95-a724-532110f24a62/volumes"
Oct 30 03:28:14.930132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:14.930032    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:28:15.068875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.068766    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"]
Oct 30 03:28:15.076927 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:28:15.076893    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e44e75_daa3_426a_9b0d_f8f8bd7c7102.slice/crio-95aa2b41122424a8ab5d31dffc115c10ffbd1fd374e920159d70dae9935424db WatchSource:0}: Error finding container 95aa2b41122424a8ab5d31dffc115c10ffbd1fd374e920159d70dae9935424db: Status 404 returned error can't find the container with id 95aa2b41122424a8ab5d31dffc115c10ffbd1fd374e920159d70dae9935424db
Oct 30 03:28:15.180881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.180780    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:15.296758 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.296685    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"]
Oct 30 03:28:15.303812 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:28:15.303724    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c110f7f_b9d7_4b4e_bf2b_8b74d602c216.slice/crio-0098cb41553a6f352f98046e9ee6401d48eabd6447713858d71067993c82e216 WatchSource:0}: Error finding container 0098cb41553a6f352f98046e9ee6401d48eabd6447713858d71067993c82e216: Status 404 returned error can't find the container with id 0098cb41553a6f352f98046e9ee6401d48eabd6447713858d71067993c82e216
Oct 30 03:28:15.401317 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.401209    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x" event={"ID":"6a59ff6b-e6b4-459a-9b33-26088de3a5c9","Type":"ContainerDied","Data":"a5190085f103bbab4c05eacca11f2330bca9016b701600e00efb242f05fe21ed"}
Oct 30 03:28:15.402316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.401366    5061 scope.go:122] "RemoveContainer" containerID="e0cb695ac54f015afed6b71160ad595937269bfa787166cd0f6679f91f4788d1"
Oct 30 03:28:15.402316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.401270    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"
Oct 30 03:28:15.407450 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.407373    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j" event={"ID":"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216","Type":"ContainerStarted","Data":"0098cb41553a6f352f98046e9ee6401d48eabd6447713858d71067993c82e216"}
Oct 30 03:28:15.410827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.410752    5061 generic.go:356] "Generic (PLEG): container finished" podID="f3aae8e9-b846-4409-877b-618c0b69302b" containerID="017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c" exitCode=0
Oct 30 03:28:15.411252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.410880    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" event={"ID":"f3aae8e9-b846-4409-877b-618c0b69302b","Type":"ContainerDied","Data":"017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c"}
Oct 30 03:28:15.411447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.411330    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2" event={"ID":"f3aae8e9-b846-4409-877b-618c0b69302b","Type":"ContainerDied","Data":"854492638e4ede0ad357773fd78b742947eddb62ed64b60f615e014c7fe3dd92"}
Oct 30 03:28:15.411447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.411090    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"
Oct 30 03:28:15.415087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.414673    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv" event={"ID":"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102","Type":"ContainerStarted","Data":"19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807"}
Oct 30 03:28:15.415087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.414783    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv" event={"ID":"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102","Type":"ContainerStarted","Data":"95aa2b41122424a8ab5d31dffc115c10ffbd1fd374e920159d70dae9935424db"}
Oct 30 03:28:15.415087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.414928    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:28:15.439357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.439246    5061 scope.go:122] "RemoveContainer" containerID="017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c"
Oct 30 03:28:15.460180 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.460131    5061 scope.go:122] "RemoveContainer" containerID="017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c"
Oct 30 03:28:15.460921 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:15.460874    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c\": container with ID starting with 017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c not found: ID does not exist" containerID="017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c"
Oct 30 03:28:15.461241 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.460945    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c"} err="failed to get container status \"017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c\": rpc error: code = NotFound desc = could not find container \"017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c\": container with ID starting with 017096459edd891969d95414b4f26bda254dfe97cb2b8dc8be9f3b6489986b3c not found: ID does not exist"
Oct 30 03:28:15.468188 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.468068    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv" podStartSLOduration=1.468032454 podStartE2EDuration="1.468032454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:15.462100855 +0000 UTC m=+2840.764279221" watchObservedRunningTime="2026-10-30 03:28:15.468032454 +0000 UTC m=+2840.770210819"
Oct 30 03:28:15.502829 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.502726    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"]
Oct 30 03:28:15.508019 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.507942    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d9f3ac-openshift-default-6c75b9cb5f-jj6b2"]
Oct 30 03:28:15.537656 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.537560    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"]
Oct 30 03:28:15.542319 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.542256    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e11f03/simulator-w1-e11f03-kserve-7794b65d8d-2kh6x"]
Oct 30 03:28:15.612483 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:15.612246    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5" podUID="03805bd7-ff01-4eb0-ae47-91d75b7c3ab6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:28:16.435639 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:16.435527    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j" event={"ID":"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216","Type":"ContainerStarted","Data":"454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330"}
Oct 30 03:28:16.436777 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:16.436272    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:28:16.477331 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:16.477191    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j" podStartSLOduration=2.477149364 podStartE2EDuration="2.477149364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:16.472294448 +0000 UTC m=+2841.774472814" watchObservedRunningTime="2026-10-30 03:28:16.477149364 +0000 UTC m=+2841.779327762"
Oct 30 03:28:16.812101 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:16.811998    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:16.880504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:16.880417    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a59ff6b-e6b4-459a-9b33-26088de3a5c9" path="/var/lib/kubelet/pods/6a59ff6b-e6b4-459a-9b33-26088de3a5c9/volumes"
Oct 30 03:28:16.881627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:16.881580    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3aae8e9-b846-4409-877b-618c0b69302b" path="/var/lib/kubelet/pods/f3aae8e9-b846-4409-877b-618c0b69302b/volumes"
Oct 30 03:28:18.410772 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:18.410686    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:19.631704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:19.631668    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:19.743156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:19.743117    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:19.964607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:19.964526    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"]
Oct 30 03:28:19.969942 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:19.969923    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:28:19.987443 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:19.987407    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"]
Oct 30 03:28:19.999827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:19.994485    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cf605a\""
Oct 30 03:28:19.999827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:19.994508    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cf605a-dockercfg-f8cxj\""
Oct 30 03:28:20.034511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.034479    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf\" (UID: \"bb0624f0-1f31-469a-84b5-97b1a9c42a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:28:20.034696 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.034675    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqq9\" (UniqueName: \"kubernetes.io/projected/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-kube-api-access-dkqq9\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf\" (UID: \"bb0624f0-1f31-469a-84b5-97b1a9c42a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:28:20.081878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.081821    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"]
Oct 30 03:28:20.085778 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.085756    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:28:20.103409 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.103386    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"]
Oct 30 03:28:20.136179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.136146    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqq9\" (UniqueName: \"kubernetes.io/projected/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-kube-api-access-dkqq9\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf\" (UID: \"bb0624f0-1f31-469a-84b5-97b1a9c42a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:28:20.136306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.136234    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbp4w\" (UniqueName: \"kubernetes.io/projected/338661f3-a26f-486a-a63e-e7b10c7499e2-kube-api-access-kbp4w\") pod \"payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8\" (UID: \"338661f3-a26f-486a-a63e-e7b10c7499e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:28:20.136401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.136377    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf\" (UID: \"bb0624f0-1f31-469a-84b5-97b1a9c42a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:28:20.136472 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.136456    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338661f3-a26f-486a-a63e-e7b10c7499e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8\" (UID: \"338661f3-a26f-486a-a63e-e7b10c7499e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:28:20.137065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.137043    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf\" (UID: \"bb0624f0-1f31-469a-84b5-97b1a9c42a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:28:20.154294 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.154273    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqq9\" (UniqueName: \"kubernetes.io/projected/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-kube-api-access-dkqq9\") pod \"payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf\" (UID: \"bb0624f0-1f31-469a-84b5-97b1a9c42a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:28:20.237375 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.237303    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338661f3-a26f-486a-a63e-e7b10c7499e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8\" (UID: \"338661f3-a26f-486a-a63e-e7b10c7499e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:28:20.237497 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.237449    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbp4w\" (UniqueName: \"kubernetes.io/projected/338661f3-a26f-486a-a63e-e7b10c7499e2-kube-api-access-kbp4w\") pod \"payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8\" (UID: \"338661f3-a26f-486a-a63e-e7b10c7499e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:28:20.238043 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.238025    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338661f3-a26f-486a-a63e-e7b10c7499e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8\" (UID: \"338661f3-a26f-486a-a63e-e7b10c7499e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:28:20.255386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.255364    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbp4w\" (UniqueName: \"kubernetes.io/projected/338661f3-a26f-486a-a63e-e7b10c7499e2-kube-api-access-kbp4w\") pod \"payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8\" (UID: \"338661f3-a26f-486a-a63e-e7b10c7499e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:28:20.283489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.283465    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:28:20.400119 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.400089    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:28:20.581175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.581142    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"]
Oct 30 03:28:20.584618 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:28:20.584584    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0624f0_1f31_469a_84b5_97b1a9c42a8e.slice/crio-4c5f4ade1893d20e2f21e2fcdb59fd006dde6cd28e5433d72fc7a0253b0e0beb WatchSource:0}: Error finding container 4c5f4ade1893d20e2f21e2fcdb59fd006dde6cd28e5433d72fc7a0253b0e0beb: Status 404 returned error can't find the container with id 4c5f4ade1893d20e2f21e2fcdb59fd006dde6cd28e5433d72fc7a0253b0e0beb
Oct 30 03:28:20.676137 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.676103    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:20.701257 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:20.701231    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"]
Oct 30 03:28:20.705041 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:28:20.705017    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338661f3_a26f_486a_a63e_e7b10c7499e2.slice/crio-a3a337fe451f33774f76be8dc4493e45b9a0c20d45bb5c3f9176debc9a763adf WatchSource:0}: Error finding container a3a337fe451f33774f76be8dc4493e45b9a0c20d45bb5c3f9176debc9a763adf: Status 404 returned error can't find the container with id a3a337fe451f33774f76be8dc4493e45b9a0c20d45bb5c3f9176debc9a763adf
Oct 30 03:28:21.483014 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:21.482974    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf" event={"ID":"bb0624f0-1f31-469a-84b5-97b1a9c42a8e","Type":"ContainerStarted","Data":"5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a"}
Oct 30 03:28:21.483192 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:21.483047    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:28:21.483192 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:21.483069    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf" event={"ID":"bb0624f0-1f31-469a-84b5-97b1a9c42a8e","Type":"ContainerStarted","Data":"4c5f4ade1893d20e2f21e2fcdb59fd006dde6cd28e5433d72fc7a0253b0e0beb"}
Oct 30 03:28:21.484699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:21.484673    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8" event={"ID":"338661f3-a26f-486a-a63e-e7b10c7499e2","Type":"ContainerStarted","Data":"14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b"}
Oct 30 03:28:21.484835 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:21.484713    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8" event={"ID":"338661f3-a26f-486a-a63e-e7b10c7499e2","Type":"ContainerStarted","Data":"a3a337fe451f33774f76be8dc4493e45b9a0c20d45bb5c3f9176debc9a763adf"}
Oct 30 03:28:21.484835 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:21.484776    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:28:21.521898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:21.521819    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf" podStartSLOduration=2.52180135 podStartE2EDuration="2.52180135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:21.518112885 +0000 UTC m=+2846.820291250" watchObservedRunningTime="2026-10-30 03:28:21.52180135 +0000 UTC m=+2846.823979720"
Oct 30 03:28:21.556222 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:21.556183    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8" podStartSLOduration=1.556173271 podStartE2EDuration="1.556173271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:21.553596129 +0000 UTC m=+2846.855774494" watchObservedRunningTime="2026-10-30 03:28:21.556173271 +0000 UTC m=+2846.858351636"
Oct 30 03:28:22.177042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:22.177007    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:24.101330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:24.101297    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:25.446267 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:25.446232    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:28:25.610151 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:25.610114    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5" podUID="03805bd7-ff01-4eb0-ae47-91d75b7c3ab6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:28:25.767163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:25.767084    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:26.440689 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:26.440659    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:28:27.130253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:27.130199    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:27.452536 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:27.452417    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:28:27.624324 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:27.624286    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:27.769272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:27.769181    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:28.256233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:28.256188    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:29.574665 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:29.574634    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:31.183771 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:31.183739    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:32.494048 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:32.494015    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:28:32.494508 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:32.494381    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:28:32.791711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:32.791629    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:33.379728 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.379668    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:28:33.490473 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.490438    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-plugins-config-volume\") pod \"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6\" (UID: \"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6\") "
Oct 30 03:28:33.490735 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.490714    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-kube-api-access-vhcxv\" (UniqueName: \"kubernetes.io/projected/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-kube-api-access-vhcxv\") pod \"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6\" (UID: \"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6\") "
Oct 30 03:28:33.490910 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.490889    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-plugins-config-volume" pod "03805bd7-ff01-4eb0-ae47-91d75b7c3ab6" (UID: "03805bd7-ff01-4eb0-ae47-91d75b7c3ab6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:33.491141 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.491119    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:33.492809 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.492773    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-kube-api-access-vhcxv" pod "03805bd7-ff01-4eb0-ae47-91d75b7c3ab6" (UID: "03805bd7-ff01-4eb0-ae47-91d75b7c3ab6"). InnerVolumeSpecName "kube-api-access-vhcxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:33.568431 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.568399    5061 generic.go:356] "Generic (PLEG): container finished" podID="03805bd7-ff01-4eb0-ae47-91d75b7c3ab6" containerID="dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b" exitCode=0
Oct 30 03:28:33.568876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.568455    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"
Oct 30 03:28:33.568876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.568480    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5" event={"ID":"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6","Type":"ContainerDied","Data":"dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b"}
Oct 30 03:28:33.568876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.568539    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5" event={"ID":"03805bd7-ff01-4eb0-ae47-91d75b7c3ab6","Type":"ContainerDied","Data":"0219c0d4d063055e6b3db19f0b78fe4ff8cfd0393a4915de43d272028f67ee0a"}
Oct 30 03:28:33.568876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.568563    5061 scope.go:122] "RemoveContainer" containerID="dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b"
Oct 30 03:28:33.578554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.578537    5061 scope.go:122] "RemoveContainer" containerID="dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b"
Oct 30 03:28:33.578812 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:33.578794    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b\": container with ID starting with dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b not found: ID does not exist" containerID="dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b"
Oct 30 03:28:33.578901 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.578820    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b"} err="failed to get container status \"dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b\": rpc error: code = NotFound desc = could not find container \"dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b\": container with ID starting with dbe0b49e2f49c7ea0cede28a0b770865fe3fb8eb4960373a66dac9bb6268ef8b not found: ID does not exist"
Oct 30 03:28:33.592422 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.592401    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhcxv\" (UniqueName: \"kubernetes.io/projected/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6-kube-api-access-vhcxv\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:33.616432 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.616407    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"]
Oct 30 03:28:33.619631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:33.619608    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-4b5459-6bf7866d9f-qd6b5"]
Oct 30 03:28:34.066874 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.066818    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:34.418588 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.418049    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"]
Oct 30 03:28:34.419162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.419139    5061 cpu_manager.go:414] "containerMap: removing container" podUID="03805bd7-ff01-4eb0-ae47-91d75b7c3ab6" containerName="payload-pre-processing"
Oct 30 03:28:34.419303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.419286    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03805bd7-ff01-4eb0-ae47-91d75b7c3ab6" containerName="payload-pre-processing"
Oct 30 03:28:34.419643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.419621    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="03805bd7-ff01-4eb0-ae47-91d75b7c3ab6" containerName="payload-pre-processing"
Oct 30 03:28:34.437201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.437175    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.444418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.444392    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"]
Oct 30 03:28:34.450564 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.450533    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3f72d5-openshift-default-dockercfg-dqq72\""
Oct 30 03:28:34.502266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.502238    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-socket\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.502370 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.502273    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491108be-282e-45ef-ac58-a9404fc21748-istiod-ca-cert\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.502370 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.502304    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-credential-socket\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.502462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.502388    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-istio-token\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.502506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.502476    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491108be-282e-45ef-ac58-a9404fc21748-istio-podinfo\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.502556 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.502538    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-certs\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.502624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.502607    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-envoy\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.502665 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.502650    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-data\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.502760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.502746    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2smg\" (UniqueName: \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-kube-api-access-p2smg\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.503941 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.503920    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:34.603651 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.603622    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-certs\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.603676    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-envoy\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.603717    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-data\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.603768    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2smg\" (UniqueName: \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-kube-api-access-p2smg\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.603880    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-socket\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.603914    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491108be-282e-45ef-ac58-a9404fc21748-istiod-ca-cert\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.603963    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-credential-socket\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.603991    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-istio-token\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.604029    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491108be-282e-45ef-ac58-a9404fc21748-istio-podinfo\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.604150    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-certs\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.604359    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-data\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.604431    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-credential-socket\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604618 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.604451    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-socket\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.604867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.604817    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491108be-282e-45ef-ac58-a9404fc21748-istiod-ca-cert\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.606192 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.606174    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-envoy\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.606563 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.606545    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491108be-282e-45ef-ac58-a9404fc21748-istio-podinfo\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.622721 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.622693    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2smg\" (UniqueName: \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-kube-api-access-p2smg\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.623343 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.623324    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-istio-token\") pod \"e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") " pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.653743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.653712    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:34.769330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.769246    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:34.850728 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:34.850695    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03805bd7-ff01-4eb0-ae47-91d75b7c3ab6" path="/var/lib/kubelet/pods/03805bd7-ff01-4eb0-ae47-91d75b7c3ab6/volumes"
Oct 30 03:28:35.088423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.088400    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"]
Oct 30 03:28:35.091327 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:28:35.091302    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491108be_282e_45ef_ac58_a9404fc21748.slice/crio-bbc7ab43723328dcf1c90de42ad22c3a7436f3a138c7081c8a5173a21d8ec1af WatchSource:0}: Error finding container bbc7ab43723328dcf1c90de42ad22c3a7436f3a138c7081c8a5173a21d8ec1af: Status 404 returned error can't find the container with id bbc7ab43723328dcf1c90de42ad22c3a7436f3a138c7081c8a5173a21d8ec1af
Oct 30 03:28:35.093729 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.093696    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:28:35.093803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.093759    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:28:35.093803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.093795    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:28:35.391736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.391644    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:35.515345 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.515301    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:35.582943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.582908    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" event={"ID":"491108be-282e-45ef-ac58-a9404fc21748","Type":"ContainerStarted","Data":"84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227"}
Oct 30 03:28:35.583110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.582990    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" event={"ID":"491108be-282e-45ef-ac58-a9404fc21748","Type":"ContainerStarted","Data":"bbc7ab43723328dcf1c90de42ad22c3a7436f3a138c7081c8a5173a21d8ec1af"}
Oct 30 03:28:35.671507 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.671405    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" podStartSLOduration=1.67139215 podStartE2EDuration="1.67139215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:35.66521579 +0000 UTC m=+2860.967394154" watchObservedRunningTime="2026-10-30 03:28:35.67139215 +0000 UTC m=+2860.973570514"
Oct 30 03:28:35.758441 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.758406    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"]
Oct 30 03:28:35.758669 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.758638    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" podUID="d05c5ab7-4d93-4f89-8207-d440fffedf22" containerName="istio-proxy" containerID="cri-o://a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d" gracePeriod=30
Oct 30 03:28:35.769523 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.769500    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:35.811018 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.810993    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:35.944495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:35.944400    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:36.204134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:36.204038    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:36.570622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:36.570590    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:36.775209 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:36.775177    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:36.782663 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:36.782638    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:37.066085 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:37.066058    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:37.595778 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:37.595750    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:37.596678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:37.596654    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:28:37.781768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:37.781727    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:38.338658 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:38.338626    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:38.456609 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:38.456578    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:38.930797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:38.930767    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:40.099015 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:40.098980    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:40.223274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:40.223240    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:41.036202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.036139    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:41.042444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.042422    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:28:41.095981 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.095945    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-socket\") pod \"d05c5ab7-4d93-4f89-8207-d440fffedf22\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") "
Oct 30 03:28:41.096172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096069    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-certs\") pod \"d05c5ab7-4d93-4f89-8207-d440fffedf22\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") "
Oct 30 03:28:41.096172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096099    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-podinfo\") pod \"d05c5ab7-4d93-4f89-8207-d440fffedf22\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") "
Oct 30 03:28:41.096291 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096177    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-envoy\") pod \"d05c5ab7-4d93-4f89-8207-d440fffedf22\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") "
Oct 30 03:28:41.096291 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096229    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05c5ab7-4d93-4f89-8207-d440fffedf22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05c5ab7-4d93-4f89-8207-d440fffedf22-istiod-ca-cert\") pod \"d05c5ab7-4d93-4f89-8207-d440fffedf22\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") "
Oct 30 03:28:41.096291 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096269    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-socket" pod "d05c5ab7-4d93-4f89-8207-d440fffedf22" (UID: "d05c5ab7-4d93-4f89-8207-d440fffedf22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:41.096455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096279    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-credential-socket\") pod \"d05c5ab7-4d93-4f89-8207-d440fffedf22\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") "
Oct 30 03:28:41.096455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096385    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-certs" pod "d05c5ab7-4d93-4f89-8207-d440fffedf22" (UID: "d05c5ab7-4d93-4f89-8207-d440fffedf22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:41.096455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096427    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-data\") pod \"d05c5ab7-4d93-4f89-8207-d440fffedf22\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") "
Oct 30 03:28:41.096603 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096477    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-token\" (UniqueName: \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-token\") pod \"d05c5ab7-4d93-4f89-8207-d440fffedf22\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") "
Oct 30 03:28:41.096603 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096507    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-kube-api-access-wlrqv\" (UniqueName: \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-kube-api-access-wlrqv\") pod \"d05c5ab7-4d93-4f89-8207-d440fffedf22\" (UID: \"d05c5ab7-4d93-4f89-8207-d440fffedf22\") "
Oct 30 03:28:41.096603 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096580    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-credential-socket" pod "d05c5ab7-4d93-4f89-8207-d440fffedf22" (UID: "d05c5ab7-4d93-4f89-8207-d440fffedf22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:41.097596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.096761    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05c5ab7-4d93-4f89-8207-d440fffedf22-istiod-ca-cert" pod "d05c5ab7-4d93-4f89-8207-d440fffedf22" (UID: "d05c5ab7-4d93-4f89-8207-d440fffedf22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:41.097596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.097194    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.097596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.097220    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.097596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.097234    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05c5ab7-4d93-4f89-8207-d440fffedf22-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.097596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.097245    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.097596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.097260    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-data" pod "d05c5ab7-4d93-4f89-8207-d440fffedf22" (UID: "d05c5ab7-4d93-4f89-8207-d440fffedf22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:41.099334 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.099294    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-envoy" pod "d05c5ab7-4d93-4f89-8207-d440fffedf22" (UID: "d05c5ab7-4d93-4f89-8207-d440fffedf22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:41.099334 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.099308    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-podinfo" pod "d05c5ab7-4d93-4f89-8207-d440fffedf22" (UID: "d05c5ab7-4d93-4f89-8207-d440fffedf22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:28:41.099776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.099441    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-token" pod "d05c5ab7-4d93-4f89-8207-d440fffedf22" (UID: "d05c5ab7-4d93-4f89-8207-d440fffedf22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:41.100672 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.100642    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-kube-api-access-wlrqv" pod "d05c5ab7-4d93-4f89-8207-d440fffedf22" (UID: "d05c5ab7-4d93-4f89-8207-d440fffedf22"). InnerVolumeSpecName "kube-api-access-wlrqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:41.198686 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.198653    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.198686 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.198684    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlrqv\" (UniqueName: \"kubernetes.io/projected/d05c5ab7-4d93-4f89-8207-d440fffedf22-kube-api-access-wlrqv\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.198902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.198694    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.198902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.198703    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.198902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.198712    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05c5ab7-4d93-4f89-8207-d440fffedf22-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:41.621243 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.621206    5061 generic.go:356] "Generic (PLEG): container finished" podID="d05c5ab7-4d93-4f89-8207-d440fffedf22" containerID="a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d" exitCode=0
Oct 30 03:28:41.621486 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.621250    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"
Oct 30 03:28:41.621486 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.621258    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" event={"ID":"d05c5ab7-4d93-4f89-8207-d440fffedf22","Type":"ContainerDied","Data":"a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d"}
Oct 30 03:28:41.621486 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.621301    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl" event={"ID":"d05c5ab7-4d93-4f89-8207-d440fffedf22","Type":"ContainerDied","Data":"86f17ab63ddf37cdca791997f94a89317c978480fead34cd901257035aa6432e"}
Oct 30 03:28:41.621486 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.621318    5061 scope.go:122] "RemoveContainer" containerID="a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d"
Oct 30 03:28:41.632784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.632760    5061 scope.go:122] "RemoveContainer" containerID="a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d"
Oct 30 03:28:41.633091 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:41.633065    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d\": container with ID starting with a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d not found: ID does not exist" containerID="a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d"
Oct 30 03:28:41.633169 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.633096    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d"} err="failed to get container status \"a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d\": rpc error: code = NotFound desc = could not find container \"a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d\": container with ID starting with a6d68e06370a2ea633edc5faeabcae1f09ce807589070248f4c60451e0be777d not found: ID does not exist"
Oct 30 03:28:41.681177 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.681141    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"]
Oct 30 03:28:41.691289 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:41.691263    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-4b5459-openshift-default-5dfb76cbfc-7hhtl"]
Oct 30 03:28:42.848418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:42.848383    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05c5ab7-4d93-4f89-8207-d440fffedf22" path="/var/lib/kubelet/pods/d05c5ab7-4d93-4f89-8207-d440fffedf22/volumes"
Oct 30 03:28:43.074888 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:43.074843    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:43.755202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:43.755164    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:43.871664 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:43.871632    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:44.515202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.515171    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:44.578617 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.578590    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"]
Oct 30 03:28:44.578817 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.578794    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4" podUID="96459a9f-427b-42d4-a568-d0a3cc7c9243" containerName="payload-processing" containerID="cri-o://0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641" gracePeriod=30
Oct 30 03:28:44.593004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.592956    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"]
Oct 30 03:28:44.593208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.593182    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b" podUID="0d8b1b1e-db85-4dde-942b-3cb7920a4738" containerName="payload-pre-processing" containerID="cri-o://00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b" gracePeriod=30
Oct 30 03:28:44.673585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.673521    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:44.799945 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.799839    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:44.871743 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.871718    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:28:44.906899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.906815    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:28:44.976717 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.976668    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96459a9f-427b-42d4-a568-d0a3cc7c9243-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96459a9f-427b-42d4-a568-d0a3cc7c9243-plugins-config-volume\") pod \"96459a9f-427b-42d4-a568-d0a3cc7c9243\" (UID: \"96459a9f-427b-42d4-a568-d0a3cc7c9243\") "
Oct 30 03:28:44.976954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.976761    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96459a9f-427b-42d4-a568-d0a3cc7c9243-kube-api-access-sdjnf\" (UniqueName: \"kubernetes.io/projected/96459a9f-427b-42d4-a568-d0a3cc7c9243-kube-api-access-sdjnf\") pod \"96459a9f-427b-42d4-a568-d0a3cc7c9243\" (UID: \"96459a9f-427b-42d4-a568-d0a3cc7c9243\") "
Oct 30 03:28:44.977460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.977427    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96459a9f-427b-42d4-a568-d0a3cc7c9243-plugins-config-volume" pod "96459a9f-427b-42d4-a568-d0a3cc7c9243" (UID: "96459a9f-427b-42d4-a568-d0a3cc7c9243"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:44.977723 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.977701    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96459a9f-427b-42d4-a568-d0a3cc7c9243-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:44.992559 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:44.992494    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96459a9f-427b-42d4-a568-d0a3cc7c9243-kube-api-access-sdjnf" pod "96459a9f-427b-42d4-a568-d0a3cc7c9243" (UID: "96459a9f-427b-42d4-a568-d0a3cc7c9243"). InnerVolumeSpecName "kube-api-access-sdjnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:45.079140 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.079032    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8b1b1e-db85-4dde-942b-3cb7920a4738-kube-api-access-r8bf8\" (UniqueName: \"kubernetes.io/projected/0d8b1b1e-db85-4dde-942b-3cb7920a4738-kube-api-access-r8bf8\") pod \"0d8b1b1e-db85-4dde-942b-3cb7920a4738\" (UID: \"0d8b1b1e-db85-4dde-942b-3cb7920a4738\") "
Oct 30 03:28:45.079140 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.079090    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8b1b1e-db85-4dde-942b-3cb7920a4738-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8b1b1e-db85-4dde-942b-3cb7920a4738-plugins-config-volume\") pod \"0d8b1b1e-db85-4dde-942b-3cb7920a4738\" (UID: \"0d8b1b1e-db85-4dde-942b-3cb7920a4738\") "
Oct 30 03:28:45.079413 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.079377    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdjnf\" (UniqueName: \"kubernetes.io/projected/96459a9f-427b-42d4-a568-d0a3cc7c9243-kube-api-access-sdjnf\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:45.079516 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.079486    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8b1b1e-db85-4dde-942b-3cb7920a4738-plugins-config-volume" pod "0d8b1b1e-db85-4dde-942b-3cb7920a4738" (UID: "0d8b1b1e-db85-4dde-942b-3cb7920a4738"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:45.081491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.081462    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8b1b1e-db85-4dde-942b-3cb7920a4738-kube-api-access-r8bf8" pod "0d8b1b1e-db85-4dde-942b-3cb7920a4738" (UID: "0d8b1b1e-db85-4dde-942b-3cb7920a4738"). InnerVolumeSpecName "kube-api-access-r8bf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:45.180836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.180780    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8bf8\" (UniqueName: \"kubernetes.io/projected/0d8b1b1e-db85-4dde-942b-3cb7920a4738-kube-api-access-r8bf8\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:45.180836 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.180830    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d8b1b1e-db85-4dde-942b-3cb7920a4738-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:45.188402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.188365    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:45.347021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.346896    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:45.656937 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.656906    5061 generic.go:356] "Generic (PLEG): container finished" podID="96459a9f-427b-42d4-a568-d0a3cc7c9243" containerID="0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641" exitCode=0
Oct 30 03:28:45.657116 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.656976    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"
Oct 30 03:28:45.657116 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.656997    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4" event={"ID":"96459a9f-427b-42d4-a568-d0a3cc7c9243","Type":"ContainerDied","Data":"0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641"}
Oct 30 03:28:45.657116 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.657083    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4" event={"ID":"96459a9f-427b-42d4-a568-d0a3cc7c9243","Type":"ContainerDied","Data":"9820f51f45b06b9ae0fe19303bbb0cabdda799bc9593a65f020267b5e766e595"}
Oct 30 03:28:45.657116 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.657110    5061 scope.go:122] "RemoveContainer" containerID="0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641"
Oct 30 03:28:45.658530 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.658504    5061 generic.go:356] "Generic (PLEG): container finished" podID="0d8b1b1e-db85-4dde-942b-3cb7920a4738" containerID="00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b" exitCode=0
Oct 30 03:28:45.658628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.658561    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"
Oct 30 03:28:45.658628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.658577    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b" event={"ID":"0d8b1b1e-db85-4dde-942b-3cb7920a4738","Type":"ContainerDied","Data":"00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b"}
Oct 30 03:28:45.658716 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.658627    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b" event={"ID":"0d8b1b1e-db85-4dde-942b-3cb7920a4738","Type":"ContainerDied","Data":"592404dac08d7ee985584461bd3d896a20860dfa82da8dfcef476c1a1fef60ec"}
Oct 30 03:28:45.669655 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.669634    5061 scope.go:122] "RemoveContainer" containerID="0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641"
Oct 30 03:28:45.670252 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:45.670226    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641\": container with ID starting with 0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641 not found: ID does not exist" containerID="0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641"
Oct 30 03:28:45.670374 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.670263    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641"} err="failed to get container status \"0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641\": rpc error: code = NotFound desc = could not find container \"0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641\": container with ID starting with 0d4a729c8a3f0e527d7fad6ce7dec84eae6d5d1fba15326e1fb45144f31af641 not found: ID does not exist"
Oct 30 03:28:45.670374 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.670287    5061 scope.go:122] "RemoveContainer" containerID="00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b"
Oct 30 03:28:45.679416 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.679399    5061 scope.go:122] "RemoveContainer" containerID="00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b"
Oct 30 03:28:45.679684 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:45.679665    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b\": container with ID starting with 00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b not found: ID does not exist" containerID="00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b"
Oct 30 03:28:45.679756 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.679689    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b"} err="failed to get container status \"00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b\": rpc error: code = NotFound desc = could not find container \"00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b\": container with ID starting with 00ae8b29c8ff00855c27f9b6af923a9f5ce77d118251da8ac994a3b9a2d4705b not found: ID does not exist"
Oct 30 03:28:45.716952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.716926    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"]
Oct 30 03:28:45.727418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.727383    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9528f-65b447dd7b-q7kv4"]
Oct 30 03:28:45.751246 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.751224    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"]
Oct 30 03:28:45.754165 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.754144    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9528f-5d6d9745b7-6tr4b"]
Oct 30 03:28:45.821930 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:45.821892    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:46.848401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:46.848367    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8b1b1e-db85-4dde-942b-3cb7920a4738" path="/var/lib/kubelet/pods/0d8b1b1e-db85-4dde-942b-3cb7920a4738/volumes"
Oct 30 03:28:46.848813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:46.848798    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96459a9f-427b-42d4-a568-d0a3cc7c9243" path="/var/lib/kubelet/pods/96459a9f-427b-42d4-a568-d0a3cc7c9243/volumes"
Oct 30 03:28:46.851134 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:46.851109    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:46.967726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:46.967693    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:48.286305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.286272    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"]
Oct 30 03:28:48.286718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.286472    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6" podUID="b9c86dfa-68e3-479a-894d-8655a536eda0" containerName="maas-api" containerID="cri-o://a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818" gracePeriod=30
Oct 30 03:28:48.374138 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.374102    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:48.583543 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.583515    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 03:28:48.634597 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.634564    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-maas-api-tls\") pod \"b9c86dfa-68e3-479a-894d-8655a536eda0\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") "
Oct 30 03:28:48.634750 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.634613    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c86dfa-68e3-479a-894d-8655a536eda0-kube-api-access-4t9tg\" (UniqueName: \"kubernetes.io/projected/b9c86dfa-68e3-479a-894d-8655a536eda0-kube-api-access-4t9tg\") pod \"b9c86dfa-68e3-479a-894d-8655a536eda0\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") "
Oct 30 03:28:48.634750 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.634697    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-metrics-certs\") pod \"b9c86dfa-68e3-479a-894d-8655a536eda0\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") "
Oct 30 03:28:48.634750 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.634721    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c86dfa-68e3-479a-894d-8655a536eda0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c86dfa-68e3-479a-894d-8655a536eda0-openshift-service-ca\") pod \"b9c86dfa-68e3-479a-894d-8655a536eda0\" (UID: \"b9c86dfa-68e3-479a-894d-8655a536eda0\") "
Oct 30 03:28:48.635219 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.635190    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c86dfa-68e3-479a-894d-8655a536eda0-openshift-service-ca" pod "b9c86dfa-68e3-479a-894d-8655a536eda0" (UID: "b9c86dfa-68e3-479a-894d-8655a536eda0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:48.636809 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.636778    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-metrics-certs" pod "b9c86dfa-68e3-479a-894d-8655a536eda0" (UID: "b9c86dfa-68e3-479a-894d-8655a536eda0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:48.637276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.637247    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c86dfa-68e3-479a-894d-8655a536eda0-kube-api-access-4t9tg" pod "b9c86dfa-68e3-479a-894d-8655a536eda0" (UID: "b9c86dfa-68e3-479a-894d-8655a536eda0"). InnerVolumeSpecName "kube-api-access-4t9tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:48.637955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.637933    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-maas-api-tls" pod "b9c86dfa-68e3-479a-894d-8655a536eda0" (UID: "b9c86dfa-68e3-479a-894d-8655a536eda0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:48.679860 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.679825    5061 generic.go:356] "Generic (PLEG): container finished" podID="b9c86dfa-68e3-479a-894d-8655a536eda0" containerID="a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818" exitCode=0
Oct 30 03:28:48.679962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.679911    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"
Oct 30 03:28:48.679962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.679922    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6" event={"ID":"b9c86dfa-68e3-479a-894d-8655a536eda0","Type":"ContainerDied","Data":"a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818"}
Oct 30 03:28:48.680106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.679990    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6" event={"ID":"b9c86dfa-68e3-479a-894d-8655a536eda0","Type":"ContainerDied","Data":"ea273549a17babab332a392953d562b19628abbdb7333be746838344a63109a7"}
Oct 30 03:28:48.680106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.680024    5061 scope.go:122] "RemoveContainer" containerID="a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818"
Oct 30 03:28:48.689725 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.689707    5061 scope.go:122] "RemoveContainer" containerID="a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818"
Oct 30 03:28:48.690024 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:48.690004    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818\": container with ID starting with a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818 not found: ID does not exist" containerID="a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818"
Oct 30 03:28:48.690097 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.690031    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818"} err="failed to get container status \"a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818\": rpc error: code = NotFound desc = could not find container \"a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818\": container with ID starting with a01ea5078ea22839dabf127b6204f33db9d0cc893bb8e015133f6aaf22b1f818 not found: ID does not exist"
Oct 30 03:28:48.725629 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.725603    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"]
Oct 30 03:28:48.731560 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.731539    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e11f03-6fb5d88bbf-phwn6"]
Oct 30 03:28:48.737113 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.737085    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:48.737223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.737119    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9tg\" (UniqueName: \"kubernetes.io/projected/b9c86dfa-68e3-479a-894d-8655a536eda0-kube-api-access-4t9tg\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:48.737223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.737137    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c86dfa-68e3-479a-894d-8655a536eda0-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:48.737223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.737153    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c86dfa-68e3-479a-894d-8655a536eda0-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:28:48.847322 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:48.847244    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c86dfa-68e3-479a-894d-8655a536eda0" path="/var/lib/kubelet/pods/b9c86dfa-68e3-479a-894d-8655a536eda0/volumes"
Oct 30 03:28:49.037808 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:49.037775    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:49.295530 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:49.295493    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:49.439719 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:49.439688    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:49.687912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:49.687810    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:50.523648 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:50.523616    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:51.014405 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:51.014370    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:52.146764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:52.146689    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:52.267388 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:52.267081    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:52.489970 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:52.489705    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:52.907440 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:52.907359    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:54.487564 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:54.487456    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:55.057050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:55.057011    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:55.273258 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:55.273165    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:55.392581 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:55.392336    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:55.472797 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:28:55.472677    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:28:55.630059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:55.629991    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:55.792441 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:55.792242    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:55.959499 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:55.959461    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"]
Oct 30 03:28:55.959792 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:55.959759    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerName="istio-proxy" containerID="cri-o://fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a" gracePeriod=30
Oct 30 03:28:56.001628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:56.001593    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:56.131052 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:56.131023    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:57.533243 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:57.533202    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:58.962643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.962612    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"]
Oct 30 03:28:58.963115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963101    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0d8b1b1e-db85-4dde-942b-3cb7920a4738" containerName="payload-pre-processing"
Oct 30 03:28:58.963166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963116    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8b1b1e-db85-4dde-942b-3cb7920a4738" containerName="payload-pre-processing"
Oct 30 03:28:58.963166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963131    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b9c86dfa-68e3-479a-894d-8655a536eda0" containerName="maas-api"
Oct 30 03:28:58.963166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963136    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c86dfa-68e3-479a-894d-8655a536eda0" containerName="maas-api"
Oct 30 03:28:58.963166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963146    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d05c5ab7-4d93-4f89-8207-d440fffedf22" containerName="istio-proxy"
Oct 30 03:28:58.963166 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963151    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05c5ab7-4d93-4f89-8207-d440fffedf22" containerName="istio-proxy"
Oct 30 03:28:58.963329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963176    5061 cpu_manager.go:414] "containerMap: removing container" podUID="96459a9f-427b-42d4-a568-d0a3cc7c9243" containerName="payload-processing"
Oct 30 03:28:58.963329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963181    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96459a9f-427b-42d4-a568-d0a3cc7c9243" containerName="payload-processing"
Oct 30 03:28:58.963329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963267    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8b1b1e-db85-4dde-942b-3cb7920a4738" containerName="payload-pre-processing"
Oct 30 03:28:58.963329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963276    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c86dfa-68e3-479a-894d-8655a536eda0" containerName="maas-api"
Oct 30 03:28:58.963329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963284    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05c5ab7-4d93-4f89-8207-d440fffedf22" containerName="istio-proxy"
Oct 30 03:28:58.963329 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.963294    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="96459a9f-427b-42d4-a568-d0a3cc7c9243" containerName="payload-processing"
Oct 30 03:28:58.968382 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.968361    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:58.985289 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:58.985267    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3f72d5-serving-cert\""
Oct 30 03:28:59.001955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.001924    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"]
Oct 30 03:28:59.069086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.069055    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:59.159058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.159022    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-metrics-certs\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.159237 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.159078    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.159316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.159248    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2fedcd9-ed61-4532-ae59-8583c533412e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.159370 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.159353    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqpl\" (UniqueName: \"kubernetes.io/projected/c2fedcd9-ed61-4532-ae59-8583c533412e-kube-api-access-clqpl\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.192929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.192901    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:59.260826 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.260724    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2fedcd9-ed61-4532-ae59-8583c533412e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.261022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.260840    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqpl\" (UniqueName: \"kubernetes.io/projected/c2fedcd9-ed61-4532-ae59-8583c533412e-kube-api-access-clqpl\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.261022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.261007    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-metrics-certs\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.261144 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.261046    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.261417 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.261390    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2fedcd9-ed61-4532-ae59-8583c533412e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.263472 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.263448    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-metrics-certs\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.263642 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.263505    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.279936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.279910    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqpl\" (UniqueName: \"kubernetes.io/projected/c2fedcd9-ed61-4532-ae59-8583c533412e-kube-api-access-clqpl\") pod \"maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.281635 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.281614    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:28:59.309106 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.309080    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:59.580627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.580594    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"]
Oct 30 03:28:59.583872 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:28:59.583822    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fedcd9_ed61_4532_ae59_8583c533412e.slice/crio-fb13a4a2d3b9a1b511ffba1a79b5cc42bd06563bd408ec2a9c93a55823b750eb WatchSource:0}: Error finding container fb13a4a2d3b9a1b511ffba1a79b5cc42bd06563bd408ec2a9c93a55823b750eb: Status 404 returned error can't find the container with id fb13a4a2d3b9a1b511ffba1a79b5cc42bd06563bd408ec2a9c93a55823b750eb
Oct 30 03:28:59.585346 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.585329    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:28:59.762705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:28:59.762664    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h" event={"ID":"c2fedcd9-ed61-4532-ae59-8583c533412e","Type":"ContainerStarted","Data":"fb13a4a2d3b9a1b511ffba1a79b5cc42bd06563bd408ec2a9c93a55823b750eb"}
Oct 30 03:29:00.770209 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:00.770167    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h" event={"ID":"c2fedcd9-ed61-4532-ae59-8583c533412e","Type":"ContainerStarted","Data":"3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76"}
Oct 30 03:29:00.770620 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:00.770233    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:29:00.808089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:00.808039    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h" podStartSLOduration=2.396745143 podStartE2EDuration="2.808026381s" totalImagesPullingTime="411.281238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:00.806827746 +0000 UTC m=+2886.109006111" watchObservedRunningTime="2026-10-30 03:29:00.808026381 +0000 UTC m=+2886.110204746"
Oct 30 03:29:00.886895 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:00.886863    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:01.225025 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.225001    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 03:29:01.304397 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304363    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-certs\") pod \"d2fa761a-333a-4f3d-b5f3-0828a4550456\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") "
Oct 30 03:29:01.304584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304405    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-socket\") pod \"d2fa761a-333a-4f3d-b5f3-0828a4550456\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") "
Oct 30 03:29:01.304584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304450    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-credential-socket\") pod \"d2fa761a-333a-4f3d-b5f3-0828a4550456\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") "
Oct 30 03:29:01.304584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304503    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-data\") pod \"d2fa761a-333a-4f3d-b5f3-0828a4550456\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") "
Oct 30 03:29:01.304975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304636    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-token\" (UniqueName: \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-token\") pod \"d2fa761a-333a-4f3d-b5f3-0828a4550456\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") "
Oct 30 03:29:01.304975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304680    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2fa761a-333a-4f3d-b5f3-0828a4550456-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fa761a-333a-4f3d-b5f3-0828a4550456-istiod-ca-cert\") pod \"d2fa761a-333a-4f3d-b5f3-0828a4550456\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") "
Oct 30 03:29:01.304975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304708    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-kube-api-access-h2b9l\" (UniqueName: \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-kube-api-access-h2b9l\") pod \"d2fa761a-333a-4f3d-b5f3-0828a4550456\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") "
Oct 30 03:29:01.304975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304714    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-certs" pod "d2fa761a-333a-4f3d-b5f3-0828a4550456" (UID: "d2fa761a-333a-4f3d-b5f3-0828a4550456"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:01.304975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304779    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-envoy\") pod \"d2fa761a-333a-4f3d-b5f3-0828a4550456\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") "
Oct 30 03:29:01.304975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304837    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-podinfo\") pod \"d2fa761a-333a-4f3d-b5f3-0828a4550456\" (UID: \"d2fa761a-333a-4f3d-b5f3-0828a4550456\") "
Oct 30 03:29:01.305313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304779    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-socket" pod "d2fa761a-333a-4f3d-b5f3-0828a4550456" (UID: "d2fa761a-333a-4f3d-b5f3-0828a4550456"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:01.305313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.304937    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-data" pod "d2fa761a-333a-4f3d-b5f3-0828a4550456" (UID: "d2fa761a-333a-4f3d-b5f3-0828a4550456"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:01.305313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.305264    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fa761a-333a-4f3d-b5f3-0828a4550456-istiod-ca-cert" pod "d2fa761a-333a-4f3d-b5f3-0828a4550456" (UID: "d2fa761a-333a-4f3d-b5f3-0828a4550456"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:01.305477 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.305426    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-credential-socket" pod "d2fa761a-333a-4f3d-b5f3-0828a4550456" (UID: "d2fa761a-333a-4f3d-b5f3-0828a4550456"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:01.305837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.305737    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.305837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.305765    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.305837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.305779    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.305837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.305794    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.305837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.305808    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fa761a-333a-4f3d-b5f3-0828a4550456-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.307540 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.307515    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-kube-api-access-h2b9l" pod "d2fa761a-333a-4f3d-b5f3-0828a4550456" (UID: "d2fa761a-333a-4f3d-b5f3-0828a4550456"). InnerVolumeSpecName "kube-api-access-h2b9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:01.307660 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.307605    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-podinfo" pod "d2fa761a-333a-4f3d-b5f3-0828a4550456" (UID: "d2fa761a-333a-4f3d-b5f3-0828a4550456"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:29:01.307927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.307904    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-token" pod "d2fa761a-333a-4f3d-b5f3-0828a4550456" (UID: "d2fa761a-333a-4f3d-b5f3-0828a4550456"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:01.307927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.307914    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-envoy" pod "d2fa761a-333a-4f3d-b5f3-0828a4550456" (UID: "d2fa761a-333a-4f3d-b5f3-0828a4550456"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:01.406712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.406656    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.406712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.406682    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2b9l\" (UniqueName: \"kubernetes.io/projected/d2fa761a-333a-4f3d-b5f3-0828a4550456-kube-api-access-h2b9l\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.406712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.406693    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.406712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.406701    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fa761a-333a-4f3d-b5f3-0828a4550456-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:01.778747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.778658    5061 generic.go:356] "Generic (PLEG): container finished" podID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerID="fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a" exitCode=0
Oct 30 03:29:01.778747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.778724    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"
Oct 30 03:29:01.779355 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.778738    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" event={"ID":"d2fa761a-333a-4f3d-b5f3-0828a4550456","Type":"ContainerDied","Data":"fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a"}
Oct 30 03:29:01.779355 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.778827    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9" event={"ID":"d2fa761a-333a-4f3d-b5f3-0828a4550456","Type":"ContainerDied","Data":"4948be2b2e2d7ce8a016eca8efb3a8af87d6976e732ac2db8d34de6c1a243df4"}
Oct 30 03:29:01.779355 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.778869    5061 scope.go:122] "RemoveContainer" containerID="fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a"
Oct 30 03:29:01.789649 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.789630    5061 scope.go:122] "RemoveContainer" containerID="fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a"
Oct 30 03:29:01.789960 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:29:01.789934    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a\": container with ID starting with fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a not found: ID does not exist" containerID="fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a"
Oct 30 03:29:01.790076 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.789968    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a"} err="failed to get container status \"fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a\": rpc error: code = NotFound desc = could not find container \"fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a\": container with ID starting with fa514f30e452b168ca9b0f98b02e57efcf4ca5b1289f3f1f105538bc93121e9a not found: ID does not exist"
Oct 30 03:29:01.826213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.826189    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"]
Oct 30 03:29:01.836750 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.836718    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e11f03-openshift-default-84c45ddbd7-mnlr9"]
Oct 30 03:29:01.871595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.871561    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:01.981192 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:01.981160    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:02.198762 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:02.198727    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:02.514510 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:02.514427    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:02.645277 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:02.645243    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:02.846346 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:02.846313    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" path="/var/lib/kubelet/pods/d2fa761a-333a-4f3d-b5f3-0828a4550456/volumes"
Oct 30 03:29:05.020425 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:05.020397    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:06.625869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:06.625825    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:06.748714 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:06.748678    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:06.784590 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:06.784561    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:29:07.577172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:07.577127    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"]
Oct 30 03:29:07.703619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:07.703516    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:08.255174 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:08.255143    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:09.775327 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:09.775298    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:11.328267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:11.328231    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:13.010598 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:13.010558    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:13.428493 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:13.428454    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:13.553249 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:13.553209    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:14.387377 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.387320    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:14.440939 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.440908    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"]
Oct 30 03:29:14.441149 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.441110    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j" podUID="8c110f7f-b9d7-4b4e-bf2b-8b74d602c216" containerName="payload-processing" containerID="cri-o://454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330" gracePeriod=30
Oct 30 03:29:14.468965 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.468937    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"]
Oct 30 03:29:14.469243 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.469210    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv" podUID="b2e44e75-daa3-426a-9b0d-f8f8bd7c7102" containerName="payload-pre-processing" containerID="cri-o://19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807" gracePeriod=30
Oct 30 03:29:14.540252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.540222    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:14.694686 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.694658    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:14.809526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.809488    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:29:14.837369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.837341    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:29:14.859292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.859262    5061 generic.go:356] "Generic (PLEG): container finished" podID="8c110f7f-b9d7-4b4e-bf2b-8b74d602c216" containerID="454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330" exitCode=0
Oct 30 03:29:14.859444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.859302    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"
Oct 30 03:29:14.859444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.859337    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j" event={"ID":"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216","Type":"ContainerDied","Data":"454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330"}
Oct 30 03:29:14.859444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.859387    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j" event={"ID":"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216","Type":"ContainerDied","Data":"0098cb41553a6f352f98046e9ee6401d48eabd6447713858d71067993c82e216"}
Oct 30 03:29:14.859444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.859416    5061 scope.go:122] "RemoveContainer" containerID="454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330"
Oct 30 03:29:14.861457 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.861350    5061 generic.go:356] "Generic (PLEG): container finished" podID="b2e44e75-daa3-426a-9b0d-f8f8bd7c7102" containerID="19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807" exitCode=0
Oct 30 03:29:14.861457 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.861392    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"
Oct 30 03:29:14.861457 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.861396    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv" event={"ID":"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102","Type":"ContainerDied","Data":"19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807"}
Oct 30 03:29:14.861457 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.861456    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv" event={"ID":"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102","Type":"ContainerDied","Data":"95aa2b41122424a8ab5d31dffc115c10ffbd1fd374e920159d70dae9935424db"}
Oct 30 03:29:14.871517 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.871499    5061 scope.go:122] "RemoveContainer" containerID="454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330"
Oct 30 03:29:14.871901 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:29:14.871836    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330\": container with ID starting with 454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330 not found: ID does not exist" containerID="454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330"
Oct 30 03:29:14.872004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.871911    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330"} err="failed to get container status \"454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330\": rpc error: code = NotFound desc = could not find container \"454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330\": container with ID starting with 454517cc43fe9f9c29d3508a109ac509783a9111f622849c8a41e0f694cd6330 not found: ID does not exist"
Oct 30 03:29:14.872004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.871935    5061 scope.go:122] "RemoveContainer" containerID="19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807"
Oct 30 03:29:14.880763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.880733    5061 scope.go:122] "RemoveContainer" containerID="19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807"
Oct 30 03:29:14.881092 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:29:14.881059    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807\": container with ID starting with 19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807 not found: ID does not exist" containerID="19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807"
Oct 30 03:29:14.881209 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.881094    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807"} err="failed to get container status \"19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807\": rpc error: code = NotFound desc = could not find container \"19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807\": container with ID starting with 19a23562dbdd96922b2b61f5e3e2291f7aee820c53841999c6150d84f6768807 not found: ID does not exist"
Oct 30 03:29:14.890434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.890402    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-plugins-config-volume\") pod \"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216\" (UID: \"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216\") "
Oct 30 03:29:14.890549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.890485    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-kube-api-access-hm4d6\" (UniqueName: \"kubernetes.io/projected/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-kube-api-access-hm4d6\") pod \"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102\" (UID: \"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102\") "
Oct 30 03:29:14.890634 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.890617    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-plugins-config-volume\") pod \"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102\" (UID: \"b2e44e75-daa3-426a-9b0d-f8f8bd7c7102\") "
Oct 30 03:29:14.890736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.890722    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-kube-api-access-v6bbm\" (UniqueName: \"kubernetes.io/projected/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-kube-api-access-v6bbm\") pod \"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216\" (UID: \"8c110f7f-b9d7-4b4e-bf2b-8b74d602c216\") "
Oct 30 03:29:14.890796 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.890766    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-plugins-config-volume" pod "8c110f7f-b9d7-4b4e-bf2b-8b74d602c216" (UID: "8c110f7f-b9d7-4b4e-bf2b-8b74d602c216"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:14.891402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.891205    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-plugins-config-volume" pod "b2e44e75-daa3-426a-9b0d-f8f8bd7c7102" (UID: "b2e44e75-daa3-426a-9b0d-f8f8bd7c7102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:14.891402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.891352    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:14.891402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.891375    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:14.892950 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.892840    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-kube-api-access-hm4d6" pod "b2e44e75-daa3-426a-9b0d-f8f8bd7c7102" (UID: "b2e44e75-daa3-426a-9b0d-f8f8bd7c7102"). InnerVolumeSpecName "kube-api-access-hm4d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:14.893682 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.893647    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-kube-api-access-v6bbm" pod "8c110f7f-b9d7-4b4e-bf2b-8b74d602c216" (UID: "8c110f7f-b9d7-4b4e-bf2b-8b74d602c216"). InnerVolumeSpecName "kube-api-access-v6bbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:14.936933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.936903    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:14.993031 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.992998    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4d6\" (UniqueName: \"kubernetes.io/projected/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102-kube-api-access-hm4d6\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:14.993031 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:14.993027    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6bbm\" (UniqueName: \"kubernetes.io/projected/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216-kube-api-access-v6bbm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:15.067651 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:15.067574    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:15.189219 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:15.189189    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:15.202235 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:15.202202    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"]
Oct 30 03:29:15.208017 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:15.207990    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-055daa-6b6cd4ccf6-dsp8j"]
Oct 30 03:29:15.245260 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:15.245228    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"]
Oct 30 03:29:15.249613 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:15.249581    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-055daa-6cb9d6b46c-s8mqv"]
Oct 30 03:29:15.547018 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:15.546985    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:16.794243 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:16.794206    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:16.859021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:16.858981    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c110f7f-b9d7-4b4e-bf2b-8b74d602c216" path="/var/lib/kubelet/pods/8c110f7f-b9d7-4b4e-bf2b-8b74d602c216/volumes"
Oct 30 03:29:16.859438 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:16.859424    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e44e75-daa3-426a-9b0d-f8f8bd7c7102" path="/var/lib/kubelet/pods/b2e44e75-daa3-426a-9b0d-f8f8bd7c7102/volumes"
Oct 30 03:29:18.277797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:18.277763    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:19.082024 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:19.081989    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:20.031869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:20.025585    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:20.184809 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:20.184768    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:20.318548 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:20.318511    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:21.386202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:21.386163    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:21.516396 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:21.516357    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:21.718468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:21.718389    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:21.854098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:21.854061    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:22.112615 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:22.112580    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:22.598625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:22.598582    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main" containerID="cri-o://38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152" gracePeriod=45
Oct 30 03:29:22.870924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:22.870890    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 03:29:22.911262 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:22.911227    5061 generic.go:356] "Generic (PLEG): container finished" podID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerID="38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152" exitCode=0
Oct 30 03:29:22.911469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:22.911276    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"
Oct 30 03:29:22.911469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:22.911276    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" event={"ID":"459a2b31-dffd-42f3-bf06-907d543e23e0","Type":"ContainerDied","Data":"38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152"}
Oct 30 03:29:22.911469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:22.911377    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf" event={"ID":"459a2b31-dffd-42f3-bf06-907d543e23e0","Type":"ContainerDied","Data":"b6ebda5bc828b0a4b2df0d608e61c52260d33962e33b52c5b7eb258de128a1c8"}
Oct 30 03:29:22.911469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:22.911411    5061 scope.go:122] "RemoveContainer" containerID="38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152"
Oct 30 03:29:22.923310 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:22.923284    5061 scope.go:122] "RemoveContainer" containerID="38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152"
Oct 30 03:29:22.923705 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:29:22.923674    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152\": container with ID starting with 38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152 not found: ID does not exist" containerID="38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152"
Oct 30 03:29:22.923812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:22.923715    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152"} err="failed to get container status \"38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152\": rpc error: code = NotFound desc = could not find container \"38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152\": container with ID starting with 38b2cc8e2ea87040f0f4c6cad1ae551c7d4ef9144e6d4221bf5ca7e232b99152 not found: ID does not exist"
Oct 30 03:29:23.009111 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.009078    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-tmp-dir\") pod \"459a2b31-dffd-42f3-bf06-907d543e23e0\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") "
Oct 30 03:29:23.009357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.009164    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459a2b31-dffd-42f3-bf06-907d543e23e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/459a2b31-dffd-42f3-bf06-907d543e23e0-tls-certs\") pod \"459a2b31-dffd-42f3-bf06-907d543e23e0\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") "
Oct 30 03:29:23.009357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.009216    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459a2b31-dffd-42f3-bf06-907d543e23e0-kube-api-access-kwcxg\" (UniqueName: \"kubernetes.io/projected/459a2b31-dffd-42f3-bf06-907d543e23e0-kube-api-access-kwcxg\") pod \"459a2b31-dffd-42f3-bf06-907d543e23e0\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") "
Oct 30 03:29:23.009357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.009246    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-home\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-home\") pod \"459a2b31-dffd-42f3-bf06-907d543e23e0\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") "
Oct 30 03:29:23.009357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.009355    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-dshm\") pod \"459a2b31-dffd-42f3-bf06-907d543e23e0\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") "
Oct 30 03:29:23.009580 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.009434    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-model-cache\") pod \"459a2b31-dffd-42f3-bf06-907d543e23e0\" (UID: \"459a2b31-dffd-42f3-bf06-907d543e23e0\") "
Oct 30 03:29:23.009580 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.009438    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-tmp-dir" pod "459a2b31-dffd-42f3-bf06-907d543e23e0" (UID: "459a2b31-dffd-42f3-bf06-907d543e23e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.009708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.009682    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-home" pod "459a2b31-dffd-42f3-bf06-907d543e23e0" (UID: "459a2b31-dffd-42f3-bf06-907d543e23e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.009992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.009968    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.010121 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.009996    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.010506 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.010477    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-model-cache" pod "459a2b31-dffd-42f3-bf06-907d543e23e0" (UID: "459a2b31-dffd-42f3-bf06-907d543e23e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.012501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.012477    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459a2b31-dffd-42f3-bf06-907d543e23e0-kube-api-access-kwcxg" pod "459a2b31-dffd-42f3-bf06-907d543e23e0" (UID: "459a2b31-dffd-42f3-bf06-907d543e23e0"). InnerVolumeSpecName "kube-api-access-kwcxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:23.012627 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.012603    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459a2b31-dffd-42f3-bf06-907d543e23e0-tls-certs" pod "459a2b31-dffd-42f3-bf06-907d543e23e0" (UID: "459a2b31-dffd-42f3-bf06-907d543e23e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:23.012678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.012641    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-dshm" pod "459a2b31-dffd-42f3-bf06-907d543e23e0" (UID: "459a2b31-dffd-42f3-bf06-907d543e23e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.111093 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.111007    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.111093 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.111045    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459a2b31-dffd-42f3-bf06-907d543e23e0-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.111093 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.111060    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwcxg\" (UniqueName: \"kubernetes.io/projected/459a2b31-dffd-42f3-bf06-907d543e23e0-kube-api-access-kwcxg\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.111093 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.111075    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459a2b31-dffd-42f3-bf06-907d543e23e0-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.254038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.254005    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"]
Oct 30 03:29:23.256622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.256597    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-32bad3/distinct-w6-32bad3-kserve-64b6968fbb-qqwvf"]
Oct 30 03:29:23.596385 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.596342    5061 patch_prober.go:29] interesting pod/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:29:23.596602 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.596410    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:29:23.815735 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:23.815700    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:24.846992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:24.846956    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" path="/var/lib/kubelet/pods/459a2b31-dffd-42f3-bf06-907d543e23e0/volumes"
Oct 30 03:29:24.924982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:24.924951    5061 generic.go:356] "Generic (PLEG): container finished" podID="491108be-282e-45ef-ac58-a9404fc21748" containerID="84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227" exitCode=0
Oct 30 03:29:24.925142 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:24.925021    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" event={"ID":"491108be-282e-45ef-ac58-a9404fc21748","Type":"ContainerDied","Data":"84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227"}
Oct 30 03:29:24.925441 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:24.925426    5061 scope.go:122] "RemoveContainer" containerID="84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227"
Oct 30 03:29:24.926763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:24.926739    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:24.926830 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:24.926803    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:24.926884 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:24.926832    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:25.462807 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:29:25.462762    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:29:25.665795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:25.665762    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:25.950123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:25.950088    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" event={"ID":"491108be-282e-45ef-ac58-a9404fc21748","Type":"ContainerStarted","Data":"4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536"}
Oct 30 03:29:26.769502 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:26.769458    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:29:26.769683 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:26.769630    5061 patch_prober.go:29] interesting pod/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 03:29:26.769729 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:26.769683    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 03:29:27.274611 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:27.274571    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:27.769566 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:27.769517    5061 patch_prober.go:29] interesting pod/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 03:29:27.769762 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:27.769601    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 03:29:28.769748 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:28.769711    5061 patch_prober.go:29] interesting pod/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 03:29:28.770222 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:28.769790    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 03:29:28.776441 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:28.776411    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:29.770071 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:29.770033    5061 patch_prober.go:29] interesting pod/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 03:29:29.770526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:29.770093    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 03:29:30.515550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:30.515515    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:30.769880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:30.769767    5061 patch_prober.go:29] interesting pod/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 30 03:29:30.769880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:30.769823    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 30 03:29:31.775236 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:31.775199    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:29:31.775760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:31.775733    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:29:31.776482 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:31.776436    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:29:32.110468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:32.110435    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:32.786261 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:32.786230    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:32.904443 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:32.904413    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:33.286203 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.286169    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:33.448571 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.448536    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:33.676962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.676914    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"]
Oct 30 03:29:33.677460 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677447    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerName="istio-proxy"
Oct 30 03:29:33.677512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677462    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerName="istio-proxy"
Oct 30 03:29:33.677512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677476    5061 cpu_manager.go:414] "containerMap: removing container" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main"
Oct 30 03:29:33.677512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677481    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main"
Oct 30 03:29:33.677512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677495    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8c110f7f-b9d7-4b4e-bf2b-8b74d602c216" containerName="payload-processing"
Oct 30 03:29:33.677512 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677501    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c110f7f-b9d7-4b4e-bf2b-8b74d602c216" containerName="payload-processing"
Oct 30 03:29:33.677675 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677519    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b2e44e75-daa3-426a-9b0d-f8f8bd7c7102" containerName="payload-pre-processing"
Oct 30 03:29:33.677675 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677524    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e44e75-daa3-426a-9b0d-f8f8bd7c7102" containerName="payload-pre-processing"
Oct 30 03:29:33.677675 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677604    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2fa761a-333a-4f3d-b5f3-0828a4550456" containerName="istio-proxy"
Oct 30 03:29:33.677675 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677612    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="459a2b31-dffd-42f3-bf06-907d543e23e0" containerName="main"
Oct 30 03:29:33.677675 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677619    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c110f7f-b9d7-4b4e-bf2b-8b74d602c216" containerName="payload-processing"
Oct 30 03:29:33.677675 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.677627    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e44e75-daa3-426a-9b0d-f8f8bd7c7102" containerName="payload-pre-processing"
Oct 30 03:29:33.679998 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.679974    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:29:33.706800 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.706760    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3f72d5\""
Oct 30 03:29:33.711028 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.710998    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"]
Oct 30 03:29:33.711189 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.711174    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3f72d5-dockercfg-2ds7h\""
Oct 30 03:29:33.738137 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.738106    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565\" (UID: \"b08d7a98-4183-4f36-918d-3a5df4aaa0ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:29:33.738315 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.738246    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7r86\" (UniqueName: \"kubernetes.io/projected/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-kube-api-access-w7r86\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565\" (UID: \"b08d7a98-4183-4f36-918d-3a5df4aaa0ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:29:33.839462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.839422    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7r86\" (UniqueName: \"kubernetes.io/projected/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-kube-api-access-w7r86\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565\" (UID: \"b08d7a98-4183-4f36-918d-3a5df4aaa0ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:29:33.839899 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.839588    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565\" (UID: \"b08d7a98-4183-4f36-918d-3a5df4aaa0ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:29:33.840327 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.840306    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565\" (UID: \"b08d7a98-4183-4f36-918d-3a5df4aaa0ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:29:33.850643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.850616    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"]
Oct 30 03:29:33.853487 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.853472    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:29:33.860078 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.860058    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7r86\" (UniqueName: \"kubernetes.io/projected/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-kube-api-access-w7r86\") pod \"payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565\" (UID: \"b08d7a98-4183-4f36-918d-3a5df4aaa0ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:29:33.873014 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.872992    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"]
Oct 30 03:29:33.941081 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.941020    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a5a139-8b79-485f-9392-4fd1b0f18561-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk\" (UID: \"e3a5a139-8b79-485f-9392-4fd1b0f18561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:29:33.941208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.941142    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w55k\" (UniqueName: \"kubernetes.io/projected/e3a5a139-8b79-485f-9392-4fd1b0f18561-kube-api-access-6w55k\") pod \"payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk\" (UID: \"e3a5a139-8b79-485f-9392-4fd1b0f18561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:29:33.993570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:33.993534    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:29:34.042589 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:34.042555    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w55k\" (UniqueName: \"kubernetes.io/projected/e3a5a139-8b79-485f-9392-4fd1b0f18561-kube-api-access-6w55k\") pod \"payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk\" (UID: \"e3a5a139-8b79-485f-9392-4fd1b0f18561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:29:34.042737 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:34.042669    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a5a139-8b79-485f-9392-4fd1b0f18561-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk\" (UID: \"e3a5a139-8b79-485f-9392-4fd1b0f18561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:29:34.043378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:34.043353    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a5a139-8b79-485f-9392-4fd1b0f18561-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk\" (UID: \"e3a5a139-8b79-485f-9392-4fd1b0f18561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:29:34.063022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:34.062959    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w55k\" (UniqueName: \"kubernetes.io/projected/e3a5a139-8b79-485f-9392-4fd1b0f18561-kube-api-access-6w55k\") pod \"payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk\" (UID: \"e3a5a139-8b79-485f-9392-4fd1b0f18561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:29:34.177607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:34.177572    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:29:34.299804 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:34.299773    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"]
Oct 30 03:29:34.303499 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:29:34.303466    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08d7a98_4183_4f36_918d_3a5df4aaa0ab.slice/crio-707e597ef7b884186dd7086ab6ec8e11a55dd87d4ccd57f57f6357e359d843c2 WatchSource:0}: Error finding container 707e597ef7b884186dd7086ab6ec8e11a55dd87d4ccd57f57f6357e359d843c2: Status 404 returned error can't find the container with id 707e597ef7b884186dd7086ab6ec8e11a55dd87d4ccd57f57f6357e359d843c2
Oct 30 03:29:34.362987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:34.362946    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:34.483215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:34.483133    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"]
Oct 30 03:29:34.487554 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:29:34.487519    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a5a139_8b79_485f_9392_4fd1b0f18561.slice/crio-ea8a2501352c36eaf7bb6064ff7912107518eba421c7c545059dc81fc19d26a7 WatchSource:0}: Error finding container ea8a2501352c36eaf7bb6064ff7912107518eba421c7c545059dc81fc19d26a7: Status 404 returned error can't find the container with id ea8a2501352c36eaf7bb6064ff7912107518eba421c7c545059dc81fc19d26a7
Oct 30 03:29:35.019655 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:35.019608    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk" event={"ID":"e3a5a139-8b79-485f-9392-4fd1b0f18561","Type":"ContainerStarted","Data":"6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00"}
Oct 30 03:29:35.020162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:35.019678    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk" event={"ID":"e3a5a139-8b79-485f-9392-4fd1b0f18561","Type":"ContainerStarted","Data":"ea8a2501352c36eaf7bb6064ff7912107518eba421c7c545059dc81fc19d26a7"}
Oct 30 03:29:35.020162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:35.019998    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:29:35.021749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:35.021722    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565" event={"ID":"b08d7a98-4183-4f36-918d-3a5df4aaa0ab","Type":"ContainerStarted","Data":"1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4"}
Oct 30 03:29:35.021911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:35.021763    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565" event={"ID":"b08d7a98-4183-4f36-918d-3a5df4aaa0ab","Type":"ContainerStarted","Data":"707e597ef7b884186dd7086ab6ec8e11a55dd87d4ccd57f57f6357e359d843c2"}
Oct 30 03:29:35.021911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:35.021879    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:29:35.055131 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:35.055077    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk" podStartSLOduration=2.055060059 podStartE2EDuration="2.055060059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:35.053827582 +0000 UTC m=+2920.356005951" watchObservedRunningTime="2026-10-30 03:29:35.055060059 +0000 UTC m=+2920.357238427"
Oct 30 03:29:35.090038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:35.089994    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565" podStartSLOduration=2.089981107 podStartE2EDuration="2.089981107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:35.089017721 +0000 UTC m=+2920.391196087" watchObservedRunningTime="2026-10-30 03:29:35.089981107 +0000 UTC m=+2920.392159471"
Oct 30 03:29:36.034360 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:36.034321    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:37.823624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:37.823587    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:38.337716 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:38.337685    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:38.489796 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:38.489759    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:38.608184 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:38.608067    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:39.014411 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:39.014304    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:40.203883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:40.203832    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:41.653450 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:41.653419    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:43.260468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:43.260432    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:44.433926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.433888    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"]
Oct 30 03:29:44.434472 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.434093    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7" podUID="39ff9afe-ec5c-4de9-8e0b-be98f8ccc005" containerName="maas-api" containerID="cri-o://c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d" gracePeriod=30
Oct 30 03:29:44.521700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.521660    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:44.726886 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.726862    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 03:29:44.914426 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.914386    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-openshift-service-ca\") pod \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") "
Oct 30 03:29:44.914739 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.914717    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-maas-api-tls\") pod \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") "
Oct 30 03:29:44.914912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.914895    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-kube-api-access-njv5j\" (UniqueName: \"kubernetes.io/projected/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-kube-api-access-njv5j\") pod \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") "
Oct 30 03:29:44.915126 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.915109    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-metrics-certs\") pod \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\" (UID: \"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005\") "
Oct 30 03:29:44.916974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.916928    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-openshift-service-ca" pod "39ff9afe-ec5c-4de9-8e0b-be98f8ccc005" (UID: "39ff9afe-ec5c-4de9-8e0b-be98f8ccc005"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:44.923758 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.923730    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-metrics-certs" pod "39ff9afe-ec5c-4de9-8e0b-be98f8ccc005" (UID: "39ff9afe-ec5c-4de9-8e0b-be98f8ccc005"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:44.932029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.931225    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-kube-api-access-njv5j" pod "39ff9afe-ec5c-4de9-8e0b-be98f8ccc005" (UID: "39ff9afe-ec5c-4de9-8e0b-be98f8ccc005"). InnerVolumeSpecName "kube-api-access-njv5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:44.935924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.933532    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-maas-api-tls" pod "39ff9afe-ec5c-4de9-8e0b-be98f8ccc005" (UID: "39ff9afe-ec5c-4de9-8e0b-be98f8ccc005"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:44.998962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:44.998890    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:45.017038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.017011    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:45.017038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.017035    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:45.017207 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.017045    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njv5j\" (UniqueName: \"kubernetes.io/projected/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-kube-api-access-njv5j\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:45.017207 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.017056    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:45.079766 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.079737    5061 generic.go:356] "Generic (PLEG): container finished" podID="39ff9afe-ec5c-4de9-8e0b-be98f8ccc005" containerID="c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d" exitCode=0
Oct 30 03:29:45.079948 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.079782    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"
Oct 30 03:29:45.079948 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.079832    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7" event={"ID":"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005","Type":"ContainerDied","Data":"c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d"}
Oct 30 03:29:45.079948 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.079909    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7" event={"ID":"39ff9afe-ec5c-4de9-8e0b-be98f8ccc005","Type":"ContainerDied","Data":"ca40b1a7b893bfd60c181d2704800cf451f925591d58ac6a6ae5dac91e5f00ea"}
Oct 30 03:29:45.079948 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.079932    5061 scope.go:122] "RemoveContainer" containerID="c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d"
Oct 30 03:29:45.090133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.090110    5061 scope.go:122] "RemoveContainer" containerID="c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d"
Oct 30 03:29:45.090386 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:29:45.090364    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d\": container with ID starting with c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d not found: ID does not exist" containerID="c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d"
Oct 30 03:29:45.090447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.090392    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d"} err="failed to get container status \"c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d\": rpc error: code = NotFound desc = could not find container \"c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d\": container with ID starting with c751a9a7dbead9dd6cb32da6411602b5a0416607f4518760bcd398a574ce9a4d not found: ID does not exist"
Oct 30 03:29:45.128168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.128140    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"]
Oct 30 03:29:45.132769 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.132749    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:45.138842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.138820    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-32bad3-7c9548f555-k2mf7"]
Oct 30 03:29:45.567528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.567496    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:45.927562 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:45.927471    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:46.030808 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:46.030778    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:29:46.031141 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:46.031120    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:29:46.059210 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:46.059184    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:46.262168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:46.262074    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:46.396792 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:46.396747    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:46.547915 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:46.547878    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"]
Oct 30 03:29:46.548200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:46.548173    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" podUID="04e26344-2d16-4f79-8587-9291286d514e" containerName="istio-proxy" containerID="cri-o://08903f1fae4401e641a9c6151cc2da3004e2b3da63ce8db11008a61076cf4599" gracePeriod=30
Oct 30 03:29:46.549628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:46.549597    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:46.688624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:46.688334    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:46.851892 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:46.851762    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ff9afe-ec5c-4de9-8e0b-be98f8ccc005" path="/var/lib/kubelet/pods/39ff9afe-ec5c-4de9-8e0b-be98f8ccc005/volumes"
Oct 30 03:29:47.030705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:47.030674    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:49.213858 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:49.213824    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:50.643139 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:50.643100    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:50.839029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:50.838998    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:51.454812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:51.454776    5061 patch_prober.go:29] interesting pod/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:29:51.455050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:51.454837    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" podUID="04e26344-2d16-4f79-8587-9291286d514e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:29:51.547867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:51.547809    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:51.670995 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:51.670945    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:51.770737 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:51.770659    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"]
Oct 30 03:29:51.770969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:51.770937    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy" containerID="cri-o://7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6" gracePeriod=30
Oct 30 03:29:51.826028 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:51.825986    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:52.136885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.133725    5061 generic.go:356] "Generic (PLEG): container finished" podID="04e26344-2d16-4f79-8587-9291286d514e" containerID="08903f1fae4401e641a9c6151cc2da3004e2b3da63ce8db11008a61076cf4599" exitCode=0
Oct 30 03:29:52.136885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.134028    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" event={"ID":"04e26344-2d16-4f79-8587-9291286d514e","Type":"ContainerDied","Data":"08903f1fae4401e641a9c6151cc2da3004e2b3da63ce8db11008a61076cf4599"}
Oct 30 03:29:52.204721 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.204699    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:29:52.303465 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.303432    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-envoy\") pod \"04e26344-2d16-4f79-8587-9291286d514e\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") "
Oct 30 03:29:52.303648 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.303492    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04e26344-2d16-4f79-8587-9291286d514e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e26344-2d16-4f79-8587-9291286d514e-istio-podinfo\") pod \"04e26344-2d16-4f79-8587-9291286d514e\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") "
Oct 30 03:29:52.303648 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.303584    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-credential-socket\") pod \"04e26344-2d16-4f79-8587-9291286d514e\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") "
Oct 30 03:29:52.303648 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.303631    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-istio-token\" (UniqueName: \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-istio-token\") pod \"04e26344-2d16-4f79-8587-9291286d514e\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") "
Oct 30 03:29:52.303839 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.303690    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e26344-2d16-4f79-8587-9291286d514e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e26344-2d16-4f79-8587-9291286d514e-istiod-ca-cert\") pod \"04e26344-2d16-4f79-8587-9291286d514e\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") "
Oct 30 03:29:52.303839 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.303742    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-data\") pod \"04e26344-2d16-4f79-8587-9291286d514e\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") "
Oct 30 03:29:52.304003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.303902    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-kube-api-access-fm9c5\" (UniqueName: \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-kube-api-access-fm9c5\") pod \"04e26344-2d16-4f79-8587-9291286d514e\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") "
Oct 30 03:29:52.304003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.303968    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-socket\") pod \"04e26344-2d16-4f79-8587-9291286d514e\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") "
Oct 30 03:29:52.304113 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.304004    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-certs\") pod \"04e26344-2d16-4f79-8587-9291286d514e\" (UID: \"04e26344-2d16-4f79-8587-9291286d514e\") "
Oct 30 03:29:52.304113 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.304006    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-credential-socket" pod "04e26344-2d16-4f79-8587-9291286d514e" (UID: "04e26344-2d16-4f79-8587-9291286d514e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:52.304113 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.304033    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-data" pod "04e26344-2d16-4f79-8587-9291286d514e" (UID: "04e26344-2d16-4f79-8587-9291286d514e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:52.304281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.304152    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e26344-2d16-4f79-8587-9291286d514e-istiod-ca-cert" pod "04e26344-2d16-4f79-8587-9291286d514e" (UID: "04e26344-2d16-4f79-8587-9291286d514e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:52.304281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.304267    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-socket" pod "04e26344-2d16-4f79-8587-9291286d514e" (UID: "04e26344-2d16-4f79-8587-9291286d514e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:52.304407 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.304382    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-certs" pod "04e26344-2d16-4f79-8587-9291286d514e" (UID: "04e26344-2d16-4f79-8587-9291286d514e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:52.304643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.304615    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.304783 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.304646    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e26344-2d16-4f79-8587-9291286d514e-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.304783 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.304662    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.304783 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.304676    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.304783 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.304691    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.305969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.305937    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-envoy" pod "04e26344-2d16-4f79-8587-9291286d514e" (UID: "04e26344-2d16-4f79-8587-9291286d514e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:52.305969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.305946    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04e26344-2d16-4f79-8587-9291286d514e-istio-podinfo" pod "04e26344-2d16-4f79-8587-9291286d514e" (UID: "04e26344-2d16-4f79-8587-9291286d514e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:29:52.306310 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.306071    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-istio-token" pod "04e26344-2d16-4f79-8587-9291286d514e" (UID: "04e26344-2d16-4f79-8587-9291286d514e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:52.306310 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.306119    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-kube-api-access-fm9c5" pod "04e26344-2d16-4f79-8587-9291286d514e" (UID: "04e26344-2d16-4f79-8587-9291286d514e"). InnerVolumeSpecName "kube-api-access-fm9c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:52.406187 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.406108    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e26344-2d16-4f79-8587-9291286d514e-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.406187 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.406140    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e26344-2d16-4f79-8587-9291286d514e-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.406187 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.406154    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.406187 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.406168    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm9c5\" (UniqueName: \"kubernetes.io/projected/04e26344-2d16-4f79-8587-9291286d514e-kube-api-access-fm9c5\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:52.529908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.529836    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:52.642372 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:52.642341    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:53.056972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:53.056937    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:53.140863 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:53.140760    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh" event={"ID":"04e26344-2d16-4f79-8587-9291286d514e","Type":"ContainerDied","Data":"08f481a9b83aef81ab31e35d637f6fc0d7eaec25f1d9c83f9564d712c1961892"}
Oct 30 03:29:53.140863 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:53.140820    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"
Oct 30 03:29:53.140863 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:53.140831    5061 scope.go:122] "RemoveContainer" containerID="08903f1fae4401e641a9c6151cc2da3004e2b3da63ce8db11008a61076cf4599"
Oct 30 03:29:53.197927 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:53.197889    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"]
Oct 30 03:29:53.209621 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:53.209585    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-6b5f19-openshift-default-85665454c-krvxh"]
Oct 30 03:29:54.465452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:54.465360    5061 patch_prober.go:29] interesting pod/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:29:54.466357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:54.465533    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:29:54.636313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:54.636275    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:54.848947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:54.848909    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e26344-2d16-4f79-8587-9291286d514e" path="/var/lib/kubelet/pods/04e26344-2d16-4f79-8587-9291286d514e/volumes"
Oct 30 03:29:55.467187 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:29:55.467101    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:29:56.083288 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:56.083203    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:57.060222 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.060145    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 03:29:57.191504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.191468    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-socket\") pod \"f654df01-c7f4-472a-a71b-6a43df523f3a\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") "
Oct 30 03:29:57.191691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.191544    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-kube-api-access-pdk2c\" (UniqueName: \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-kube-api-access-pdk2c\") pod \"f654df01-c7f4-472a-a71b-6a43df523f3a\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") "
Oct 30 03:29:57.191691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.191602    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f654df01-c7f4-472a-a71b-6a43df523f3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f654df01-c7f4-472a-a71b-6a43df523f3a-istiod-ca-cert\") pod \"f654df01-c7f4-472a-a71b-6a43df523f3a\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") "
Oct 30 03:29:57.191691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.191633    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-credential-socket\") pod \"f654df01-c7f4-472a-a71b-6a43df523f3a\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") "
Oct 30 03:29:57.191691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.191670    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-data\") pod \"f654df01-c7f4-472a-a71b-6a43df523f3a\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") "
Oct 30 03:29:57.191900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.191766    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-socket" pod "f654df01-c7f4-472a-a71b-6a43df523f3a" (UID: "f654df01-c7f4-472a-a71b-6a43df523f3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:57.191900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.191773    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-podinfo\") pod \"f654df01-c7f4-472a-a71b-6a43df523f3a\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") "
Oct 30 03:29:57.191900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.191884    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-certs\") pod \"f654df01-c7f4-472a-a71b-6a43df523f3a\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") "
Oct 30 03:29:57.192074 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.191936    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-token\" (UniqueName: \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-token\") pod \"f654df01-c7f4-472a-a71b-6a43df523f3a\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") "
Oct 30 03:29:57.192074 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.191968    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-envoy\") pod \"f654df01-c7f4-472a-a71b-6a43df523f3a\" (UID: \"f654df01-c7f4-472a-a71b-6a43df523f3a\") "
Oct 30 03:29:57.192074 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.192027    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-data" pod "f654df01-c7f4-472a-a71b-6a43df523f3a" (UID: "f654df01-c7f4-472a-a71b-6a43df523f3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:57.192239 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.192071    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f654df01-c7f4-472a-a71b-6a43df523f3a-istiod-ca-cert" pod "f654df01-c7f4-472a-a71b-6a43df523f3a" (UID: "f654df01-c7f4-472a-a71b-6a43df523f3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:57.192300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.192240    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-certs" pod "f654df01-c7f4-472a-a71b-6a43df523f3a" (UID: "f654df01-c7f4-472a-a71b-6a43df523f3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:57.192300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.192264    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-credential-socket" pod "f654df01-c7f4-472a-a71b-6a43df523f3a" (UID: "f654df01-c7f4-472a-a71b-6a43df523f3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:57.192576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.192537    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.192576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.192566    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.192792 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.192582    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f654df01-c7f4-472a-a71b-6a43df523f3a-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.192792 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.192596    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.192792 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.192610    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.193091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.193062    5061 generic.go:356] "Generic (PLEG): container finished" podID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerID="7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6" exitCode=0
Oct 30 03:29:57.193220 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.193127    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"
Oct 30 03:29:57.193220 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.193148    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" event={"ID":"f654df01-c7f4-472a-a71b-6a43df523f3a","Type":"ContainerDied","Data":"7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6"}
Oct 30 03:29:57.193220 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.193213    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp" event={"ID":"f654df01-c7f4-472a-a71b-6a43df523f3a","Type":"ContainerDied","Data":"94f573695d28c6ea800272c6029110ae23458d6620ee19df09d6a6b88f68bc5d"}
Oct 30 03:29:57.193399 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.193245    5061 scope.go:122] "RemoveContainer" containerID="7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6"
Oct 30 03:29:57.194330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.194305    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-podinfo" pod "f654df01-c7f4-472a-a71b-6a43df523f3a" (UID: "f654df01-c7f4-472a-a71b-6a43df523f3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:29:57.194451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.194341    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-kube-api-access-pdk2c" pod "f654df01-c7f4-472a-a71b-6a43df523f3a" (UID: "f654df01-c7f4-472a-a71b-6a43df523f3a"). InnerVolumeSpecName "kube-api-access-pdk2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:57.194451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.194420    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-token" pod "f654df01-c7f4-472a-a71b-6a43df523f3a" (UID: "f654df01-c7f4-472a-a71b-6a43df523f3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:57.194590 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.194463    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-envoy" pod "f654df01-c7f4-472a-a71b-6a43df523f3a" (UID: "f654df01-c7f4-472a-a71b-6a43df523f3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:57.224298 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.224273    5061 scope.go:122] "RemoveContainer" containerID="bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed"
Oct 30 03:29:57.232919 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.232898    5061 scope.go:122] "RemoveContainer" containerID="7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6"
Oct 30 03:29:57.233196 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:29:57.233170    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6\": container with ID starting with 7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6 not found: ID does not exist" containerID="7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6"
Oct 30 03:29:57.233252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.233207    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6"} err="failed to get container status \"7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6\": rpc error: code = NotFound desc = could not find container \"7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6\": container with ID starting with 7da2a2a5bb52e066dd7b1b607a3f24645b012f9cee8dded1ab2fbf55d847fac6 not found: ID does not exist"
Oct 30 03:29:57.233252 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.233225    5061 scope.go:122] "RemoveContainer" containerID="bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed"
Oct 30 03:29:57.233447 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:29:57.233429    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed\": container with ID starting with bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed not found: ID does not exist" containerID="bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed"
Oct 30 03:29:57.233503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.233456    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed"} err="failed to get container status \"bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed\": rpc error: code = NotFound desc = could not find container \"bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed\": container with ID starting with bf09c11e8daac548015533f5f892859dd6ea1bfca8ee31a863c0284a3fb222ed not found: ID does not exist"
Oct 30 03:29:57.293332 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.293293    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.293332 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.293334    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.293556 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.293350    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f654df01-c7f4-472a-a71b-6a43df523f3a-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.293556 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.293362    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdk2c\" (UniqueName: \"kubernetes.io/projected/f654df01-c7f4-472a-a71b-6a43df523f3a-kube-api-access-pdk2c\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:29:57.490728 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.490696    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"]
Oct 30 03:29:57.524679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.524647    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"]
Oct 30 03:29:57.558392 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.558364    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"]
Oct 30 03:29:57.561001 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.560978    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-32bad3-openshift-default-6cc8655c5-2dvtp"]
Oct 30 03:29:57.653142 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.653093    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:57.769583 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.769505    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:57.884542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:57.884510    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:58.846124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:58.846092    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" path="/var/lib/kubelet/pods/f654df01-c7f4-472a-a71b-6a43df523f3a/volumes"
Oct 30 03:29:59.353752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:29:59.353708    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:00.156276 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.156238    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"]
Oct 30 03:30:00.156952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.156931    5061 cpu_manager.go:414] "containerMap: removing container" podUID="39ff9afe-ec5c-4de9-8e0b-be98f8ccc005" containerName="maas-api"
Oct 30 03:30:00.157089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.156954    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ff9afe-ec5c-4de9-8e0b-be98f8ccc005" containerName="maas-api"
Oct 30 03:30:00.157089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.156969    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy"
Oct 30 03:30:00.157089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.156977    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy"
Oct 30 03:30:00.157089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.156995    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy"
Oct 30 03:30:00.157089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.157001    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy"
Oct 30 03:30:00.157089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.157071    5061 cpu_manager.go:414] "containerMap: removing container" podUID="04e26344-2d16-4f79-8587-9291286d514e" containerName="istio-proxy"
Oct 30 03:30:00.157089 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.157079    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e26344-2d16-4f79-8587-9291286d514e" containerName="istio-proxy"
Oct 30 03:30:00.157449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.157197    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e26344-2d16-4f79-8587-9291286d514e" containerName="istio-proxy"
Oct 30 03:30:00.157449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.157214    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ff9afe-ec5c-4de9-8e0b-be98f8ccc005" containerName="maas-api"
Oct 30 03:30:00.157449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.157226    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy"
Oct 30 03:30:00.157449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.157237    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f654df01-c7f4-472a-a71b-6a43df523f3a" containerName="istio-proxy"
Oct 30 03:30:00.162050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.162028    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"
Oct 30 03:30:00.172316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.172289    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"]
Oct 30 03:30:00.181193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.181164    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkr99\""
Oct 30 03:30:00.182906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.182509    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"
Oct 30 03:30:00.188156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.188132    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"]
Oct 30 03:30:00.195584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.195560    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"]
Oct 30 03:30:00.229372 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.229340    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a5c3d4-194d-4214-89bc-697164722e46-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888850-tx2rd\" (UID: \"70a5c3d4-194d-4214-89bc-697164722e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"
Oct 30 03:30:00.229657 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.229625    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9rm\" (UniqueName: \"kubernetes.io/projected/70a5c3d4-194d-4214-89bc-697164722e46-kube-api-access-rf9rm\") pod \"maas-api-key-cleanup-29888850-tx2rd\" (UID: \"70a5c3d4-194d-4214-89bc-697164722e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"
Oct 30 03:30:00.330474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.330444    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9rm\" (UniqueName: \"kubernetes.io/projected/70a5c3d4-194d-4214-89bc-697164722e46-kube-api-access-rf9rm\") pod \"maas-api-key-cleanup-29888850-tx2rd\" (UID: \"70a5c3d4-194d-4214-89bc-697164722e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"
Oct 30 03:30:00.330595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.330505    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb448\" (UniqueName: \"kubernetes.io/projected/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-kube-api-access-qb448\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj\" (UID: \"0b27eb1a-516b-4ec1-b5b4-afc8333f6984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"
Oct 30 03:30:00.330595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.330575    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj\" (UID: \"0b27eb1a-516b-4ec1-b5b4-afc8333f6984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"
Oct 30 03:30:00.330754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.330732    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a5c3d4-194d-4214-89bc-697164722e46-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888850-tx2rd\" (UID: \"70a5c3d4-194d-4214-89bc-697164722e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"
Oct 30 03:30:00.331338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.331319    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a5c3d4-194d-4214-89bc-697164722e46-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888850-tx2rd\" (UID: \"70a5c3d4-194d-4214-89bc-697164722e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"
Oct 30 03:30:00.347602 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.347575    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9rm\" (UniqueName: \"kubernetes.io/projected/70a5c3d4-194d-4214-89bc-697164722e46-kube-api-access-rf9rm\") pod \"maas-api-key-cleanup-29888850-tx2rd\" (UID: \"70a5c3d4-194d-4214-89bc-697164722e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"
Oct 30 03:30:00.433037 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.432958    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb448\" (UniqueName: \"kubernetes.io/projected/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-kube-api-access-qb448\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj\" (UID: \"0b27eb1a-516b-4ec1-b5b4-afc8333f6984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"
Oct 30 03:30:00.433202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.433070    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj\" (UID: \"0b27eb1a-516b-4ec1-b5b4-afc8333f6984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"
Oct 30 03:30:00.433731 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.433707    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj\" (UID: \"0b27eb1a-516b-4ec1-b5b4-afc8333f6984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"
Oct 30 03:30:00.451062 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.451037    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb448\" (UniqueName: \"kubernetes.io/projected/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-kube-api-access-qb448\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj\" (UID: \"0b27eb1a-516b-4ec1-b5b4-afc8333f6984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"
Oct 30 03:30:00.485171 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.485149    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"
Oct 30 03:30:00.508030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.508001    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"
Oct 30 03:30:00.785452 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.785377    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"]
Oct 30 03:30:00.789328 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:30:00.789288    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a5c3d4_194d_4214_89bc_697164722e46.slice/crio-cb66cbe0affa59b5c9839aaf6a69a3e8c39e9cbdff33f1fa55fb4aebbc199903 WatchSource:0}: Error finding container cb66cbe0affa59b5c9839aaf6a69a3e8c39e9cbdff33f1fa55fb4aebbc199903: Status 404 returned error can't find the container with id cb66cbe0affa59b5c9839aaf6a69a3e8c39e9cbdff33f1fa55fb4aebbc199903
Oct 30 03:30:00.823794 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.823769    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"]
Oct 30 03:30:00.827241 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:30:00.827218    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b27eb1a_516b_4ec1_b5b4_afc8333f6984.slice/crio-be4161a6eaf1e8e6347989e1b0e4e851936db12e0ca7678f57c53f29786b56ba WatchSource:0}: Error finding container be4161a6eaf1e8e6347989e1b0e4e851936db12e0ca7678f57c53f29786b56ba: Status 404 returned error can't find the container with id be4161a6eaf1e8e6347989e1b0e4e851936db12e0ca7678f57c53f29786b56ba
Oct 30 03:30:00.917554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:00.917524    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:01.217994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:01.217960    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj" event={"ID":"0b27eb1a-516b-4ec1-b5b4-afc8333f6984","Type":"ContainerStarted","Data":"be4161a6eaf1e8e6347989e1b0e4e851936db12e0ca7678f57c53f29786b56ba"}
Oct 30 03:30:01.219598 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:01.219565    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd" event={"ID":"70a5c3d4-194d-4214-89bc-697164722e46","Type":"ContainerStarted","Data":"c07be992d9df44cc640f2f7ee381f5e817e7eab41aa83d085520a4f376737111"}
Oct 30 03:30:01.219755 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:01.219619    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd" event={"ID":"70a5c3d4-194d-4214-89bc-697164722e46","Type":"ContainerStarted","Data":"cb66cbe0affa59b5c9839aaf6a69a3e8c39e9cbdff33f1fa55fb4aebbc199903"}
Oct 30 03:30:01.253750 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:01.253706    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd" podStartSLOduration=1.253693995 podStartE2EDuration="1.253693995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.253220668 +0000 UTC m=+2946.555399034" watchObservedRunningTime="2026-10-30 03:30:01.253693995 +0000 UTC m=+2946.555872360"
Oct 30 03:30:02.226816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:02.226780    5061 generic.go:356] "Generic (PLEG): container finished" podID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerID="0d824e395c0378383156a0707686263fde956f7a876cdc97185636fe2224efe6" exitCode=60
Oct 30 03:30:02.227287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:02.226815    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj" event={"ID":"0b27eb1a-516b-4ec1-b5b4-afc8333f6984","Type":"ContainerDied","Data":"0d824e395c0378383156a0707686263fde956f7a876cdc97185636fe2224efe6"}
Oct 30 03:30:02.227287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:02.227136    5061 scope.go:122] "RemoveContainer" containerID="0d824e395c0378383156a0707686263fde956f7a876cdc97185636fe2224efe6"
Oct 30 03:30:02.228526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:02.228501    5061 generic.go:356] "Generic (PLEG): container finished" podID="70a5c3d4-194d-4214-89bc-697164722e46" containerID="c07be992d9df44cc640f2f7ee381f5e817e7eab41aa83d085520a4f376737111" exitCode=60
Oct 30 03:30:02.228629 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:02.228596    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd" event={"ID":"70a5c3d4-194d-4214-89bc-697164722e46","Type":"ContainerDied","Data":"c07be992d9df44cc640f2f7ee381f5e817e7eab41aa83d085520a4f376737111"}
Oct 30 03:30:02.228760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:02.228747    5061 scope.go:122] "RemoveContainer" containerID="c07be992d9df44cc640f2f7ee381f5e817e7eab41aa83d085520a4f376737111"
Oct 30 03:30:02.491678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:02.491602    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm" podUID="d7a6eb60-0637-4d7c-9ab3-769ceb0cc464" containerName="payload-processing" containerID="cri-o://c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582" gracePeriod=25
Oct 30 03:30:02.525612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:02.525576    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9" podUID="d29cfa43-cc24-4c84-a55d-12401012d345" containerName="payload-pre-processing" containerID="cri-o://1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4" gracePeriod=25
Oct 30 03:30:03.000470 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:03.000440    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:03.236779 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:03.236750    5061 generic.go:356] "Generic (PLEG): container finished" podID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerID="bbfdd030803d121d5d8cf1474e6793471d549d15f164001ff3dac858709de9ec" exitCode=60
Oct 30 03:30:03.237194 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:03.236836    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj" event={"ID":"0b27eb1a-516b-4ec1-b5b4-afc8333f6984","Type":"ContainerDied","Data":"bbfdd030803d121d5d8cf1474e6793471d549d15f164001ff3dac858709de9ec"}
Oct 30 03:30:03.237194 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:03.236959    5061 scope.go:122] "RemoveContainer" containerID="0d824e395c0378383156a0707686263fde956f7a876cdc97185636fe2224efe6"
Oct 30 03:30:03.237313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:03.237236    5061 scope.go:122] "RemoveContainer" containerID="bbfdd030803d121d5d8cf1474e6793471d549d15f164001ff3dac858709de9ec"
Oct 30 03:30:03.237595 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:03.237571    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj_odh-ai-gateway-infra(0b27eb1a-516b-4ec1-b5b4-afc8333f6984)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984"
Oct 30 03:30:03.239035 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:03.239015    5061 generic.go:356] "Generic (PLEG): container finished" podID="70a5c3d4-194d-4214-89bc-697164722e46" containerID="d4d187214043c499a72af789028471e0fbc4c1696db6268a35b31406de9abcdb" exitCode=60
Oct 30 03:30:03.239112 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:03.239064    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd" event={"ID":"70a5c3d4-194d-4214-89bc-697164722e46","Type":"ContainerDied","Data":"d4d187214043c499a72af789028471e0fbc4c1696db6268a35b31406de9abcdb"}
Oct 30 03:30:03.239387 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:03.239371    5061 scope.go:122] "RemoveContainer" containerID="d4d187214043c499a72af789028471e0fbc4c1696db6268a35b31406de9abcdb"
Oct 30 03:30:03.239676 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:03.239653    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888850-tx2rd_odh-ai-gateway-infra(70a5c3d4-194d-4214-89bc-697164722e46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd" podUID="70a5c3d4-194d-4214-89bc-697164722e46"
Oct 30 03:30:03.343814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:03.343792    5061 scope.go:122] "RemoveContainer" containerID="c07be992d9df44cc640f2f7ee381f5e817e7eab41aa83d085520a4f376737111"
Oct 30 03:30:04.246671 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:04.246642    5061 scope.go:122] "RemoveContainer" containerID="bbfdd030803d121d5d8cf1474e6793471d549d15f164001ff3dac858709de9ec"
Oct 30 03:30:04.247118 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:04.246967    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj_odh-ai-gateway-infra(0b27eb1a-516b-4ec1-b5b4-afc8333f6984)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984"
Oct 30 03:30:04.248214 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:04.248192    5061 scope.go:122] "RemoveContainer" containerID="d4d187214043c499a72af789028471e0fbc4c1696db6268a35b31406de9abcdb"
Oct 30 03:30:04.248409 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:04.248392    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888850-tx2rd_odh-ai-gateway-infra(70a5c3d4-194d-4214-89bc-697164722e46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd" podUID="70a5c3d4-194d-4214-89bc-697164722e46"
Oct 30 03:30:04.555693 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:04.555658    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:04.777114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:04.777076    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8_338661f3-a26f-486a-a63e-e7b10c7499e2/payload-processing/0.log"
Oct 30 03:30:04.958531 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:04.958444    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-v8fgz_87148799-07e3-41d4-a6f0-0e4678d90202/payload-processing/0.log"
Oct 30 03:30:05.153255 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:05.153227    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-v8fgz_87148799-07e3-41d4-a6f0-0e4678d90202/payload-processing/0.log"
Oct 30 03:30:05.332335 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:05.332311    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk_e3a5a139-8b79-485f-9392-4fd1b0f18561/payload-processing/0.log"
Oct 30 03:30:05.828933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:05.828903    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"]
Oct 30 03:30:05.834123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:05.834104    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:05.855686 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:05.855662    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"]
Oct 30 03:30:05.861250 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:05.861219    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:30:05.994112 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:05.994082    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:05.994261 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:05.994121    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:05.994449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:05.994427    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftz5b\" (UniqueName: \"kubernetes.io/projected/62ce2051-8166-4b9c-94f2-2711be84b2d1-kube-api-access-ftz5b\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:06.095444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:06.095369    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:06.095444 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:06.095413    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:06.095653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:06.095544    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftz5b\" (UniqueName: \"kubernetes.io/projected/62ce2051-8166-4b9c-94f2-2711be84b2d1-kube-api-access-ftz5b\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:06.096160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:06.096139    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:06.097762 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:06.097745    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:06.118765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:06.118743    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftz5b\" (UniqueName: \"kubernetes.io/projected/62ce2051-8166-4b9c-94f2-2711be84b2d1-kube-api-access-ftz5b\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:06.147281 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:06.147259    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:06.445376 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:06.445290    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"]
Oct 30 03:30:06.449224 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:30:06.449193    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ce2051_8166_4b9c_94f2_2711be84b2d1.slice/crio-73b6eacd497b49dd47f161d3c1bc72dee457fb69984abbf73c1bdd110c3fd120 WatchSource:0}: Error finding container 73b6eacd497b49dd47f161d3c1bc72dee457fb69984abbf73c1bdd110c3fd120: Status 404 returned error can't find the container with id 73b6eacd497b49dd47f161d3c1bc72dee457fb69984abbf73c1bdd110c3fd120
Oct 30 03:30:06.713799 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:06.713712    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:07.267711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:07.267685    5061 generic.go:356] "Generic (PLEG): container finished" podID="62ce2051-8166-4b9c-94f2-2711be84b2d1" containerID="497da6529e9b2075717d02457893f3dca7206e07392e2f5500293dd62e869213" exitCode=0
Oct 30 03:30:07.267843 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:07.267747    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg" event={"ID":"62ce2051-8166-4b9c-94f2-2711be84b2d1","Type":"ContainerDied","Data":"497da6529e9b2075717d02457893f3dca7206e07392e2f5500293dd62e869213"}
Oct 30 03:30:07.267843 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:07.267783    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg" event={"ID":"62ce2051-8166-4b9c-94f2-2711be84b2d1","Type":"ContainerStarted","Data":"73b6eacd497b49dd47f161d3c1bc72dee457fb69984abbf73c1bdd110c3fd120"}
Oct 30 03:30:08.223563 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.223525    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:08.434437 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.434414    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:08.524673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.524589    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-xks-ca\") pod \"62ce2051-8166-4b9c-94f2-2711be84b2d1\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") "
Oct 30 03:30:08.524837 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.524777    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ce2051-8166-4b9c-94f2-2711be84b2d1-kube-api-access-ftz5b\" (UniqueName: \"kubernetes.io/projected/62ce2051-8166-4b9c-94f2-2711be84b2d1-kube-api-access-ftz5b\") pod \"62ce2051-8166-4b9c-94f2-2711be84b2d1\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") "
Oct 30 03:30:08.524971 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.524956    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-openshift-ca\") pod \"62ce2051-8166-4b9c-94f2-2711be84b2d1\" (UID: \"62ce2051-8166-4b9c-94f2-2711be84b2d1\") "
Oct 30 03:30:08.525344 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.525319    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-openshift-ca" pod "62ce2051-8166-4b9c-94f2-2711be84b2d1" (UID: "62ce2051-8166-4b9c-94f2-2711be84b2d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:08.526764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.526735    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-xks-ca" pod "62ce2051-8166-4b9c-94f2-2711be84b2d1" (UID: "62ce2051-8166-4b9c-94f2-2711be84b2d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:08.526924 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.526896    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ce2051-8166-4b9c-94f2-2711be84b2d1-kube-api-access-ftz5b" pod "62ce2051-8166-4b9c-94f2-2711be84b2d1" (UID: "62ce2051-8166-4b9c-94f2-2711be84b2d1"). InnerVolumeSpecName "kube-api-access-ftz5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:08.626706 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.626664    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-openshift-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:08.626706 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.626697    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce2051-8166-4b9c-94f2-2711be84b2d1-maas-api-xks-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:08.626706 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.626709    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftz5b\" (UniqueName: \"kubernetes.io/projected/62ce2051-8166-4b9c-94f2-2711be84b2d1-kube-api-access-ftz5b\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:08.658202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.658172    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:08.895879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:08.895828    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:09.020153 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:09.020089    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:09.285809 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:09.285708    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg" event={"ID":"62ce2051-8166-4b9c-94f2-2711be84b2d1","Type":"ContainerDied","Data":"73b6eacd497b49dd47f161d3c1bc72dee457fb69984abbf73c1bdd110c3fd120"}
Oct 30 03:30:09.286388 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:09.286363    5061 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b6eacd497b49dd47f161d3c1bc72dee457fb69984abbf73c1bdd110c3fd120"
Oct 30 03:30:09.286582 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:09.286567    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"
Oct 30 03:30:09.424362 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:09.424327    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:10.689205 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:10.689168    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:11.325682 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:11.325644    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"]
Oct 30 03:30:11.325887 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:11.325865    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerName="istio-proxy" containerID="cri-o://c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426" gracePeriod=30
Oct 30 03:30:12.185959 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:12.185924    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:12.240679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:12.240640    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9" podUID="d29cfa43-cc24-4c84-a55d-12401012d345" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:30:12.240894 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:12.240640    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm" podUID="d7a6eb60-0637-4d7c-9ab3-769ceb0cc464" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:30:13.868967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:13.868935    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:14.843536 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:14.843505    5061 scope.go:122] "RemoveContainer" containerID="bbfdd030803d121d5d8cf1474e6793471d549d15f164001ff3dac858709de9ec"
Oct 30 03:30:15.155041 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:15.155009    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:15.274293 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:15.274253    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:15.323925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:15.323894    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj" event={"ID":"0b27eb1a-516b-4ec1-b5b4-afc8333f6984","Type":"ContainerStarted","Data":"4ae14b969cdfe79922bc8123ac6c618e4534b5c0ba84b13035fbd46737602ea4"}
Oct 30 03:30:15.361475 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:15.361432    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj" podStartSLOduration=15.361420847 podStartE2EDuration="15.361420847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:15.360961582 +0000 UTC m=+2960.663139957" watchObservedRunningTime="2026-10-30 03:30:15.361420847 +0000 UTC m=+2960.663599219"
Oct 30 03:30:15.588293 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:15.588256    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:15.840388 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:15.840318    5061 scope.go:122] "RemoveContainer" containerID="d4d187214043c499a72af789028471e0fbc4c1696db6268a35b31406de9abcdb"
Oct 30 03:30:15.971529 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:15.971496    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:16.100891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.100804    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:16.331429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.331396    5061 generic.go:356] "Generic (PLEG): container finished" podID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerID="4ae14b969cdfe79922bc8123ac6c618e4534b5c0ba84b13035fbd46737602ea4" exitCode=60
Oct 30 03:30:16.331884 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.331469    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj" event={"ID":"0b27eb1a-516b-4ec1-b5b4-afc8333f6984","Type":"ContainerDied","Data":"4ae14b969cdfe79922bc8123ac6c618e4534b5c0ba84b13035fbd46737602ea4"}
Oct 30 03:30:16.331884 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.331538    5061 scope.go:122] "RemoveContainer" containerID="bbfdd030803d121d5d8cf1474e6793471d549d15f164001ff3dac858709de9ec"
Oct 30 03:30:16.332027 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.331922    5061 scope.go:122] "RemoveContainer" containerID="4ae14b969cdfe79922bc8123ac6c618e4534b5c0ba84b13035fbd46737602ea4"
Oct 30 03:30:16.332228 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:16.332207    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj_odh-ai-gateway-infra(0b27eb1a-516b-4ec1-b5b4-afc8333f6984)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984"
Oct 30 03:30:16.333889 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.333843    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd" event={"ID":"70a5c3d4-194d-4214-89bc-697164722e46","Type":"ContainerStarted","Data":"165493583fded6369a0314b12c9ce1df39515edeef125461b8b497e04a5dda10"}
Oct 30 03:30:16.379409 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.379382    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"]
Oct 30 03:30:16.586707 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.586679    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:30:16.610204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610144    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-certs\") pod \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") "
Oct 30 03:30:16.610330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610233    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-data\") pod \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") "
Oct 30 03:30:16.610330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610275    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-credential-socket\") pod \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") "
Oct 30 03:30:16.610330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610307    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istiod-ca-cert\") pod \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") "
Oct 30 03:30:16.610498 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610348    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-podinfo\") pod \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") "
Oct 30 03:30:16.610498 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610382    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-certs" pod "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" (UID: "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:16.610498 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610454    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-kube-api-access-tnh8g\" (UniqueName: \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-kube-api-access-tnh8g\") pod \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") "
Oct 30 03:30:16.610662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610552    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-envoy\") pod \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") "
Oct 30 03:30:16.610662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610572    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-socket\") pod \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") "
Oct 30 03:30:16.610662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610456    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-data" pod "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" (UID: "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:16.610662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610617    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-token\" (UniqueName: \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-token\") pod \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\" (UID: \"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f\") "
Oct 30 03:30:16.610662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610616    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-credential-socket" pod "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" (UID: "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:16.610933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610666    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istiod-ca-cert" pod "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" (UID: "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:16.610933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.610885    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-socket" pod "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" (UID: "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:16.611054 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.611041    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:16.611109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.611059    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:16.611109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.611068    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:16.611109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.611077    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:16.611109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.611090    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:16.612697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.612669    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-envoy" pod "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" (UID: "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:16.612816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.612718    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-kube-api-access-tnh8g" pod "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" (UID: "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f"). InnerVolumeSpecName "kube-api-access-tnh8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:16.612816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.612736    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-podinfo" pod "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" (UID: "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:30:16.612816 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.612812    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-token" pod "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" (UID: "d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:16.712681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.712641    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:16.712681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.712681    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:16.712967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.712696    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:16.712967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:16.712709    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnh8g\" (UniqueName: \"kubernetes.io/projected/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f-kube-api-access-tnh8g\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:17.342099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.342062    5061 generic.go:356] "Generic (PLEG): container finished" podID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerID="c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426" exitCode=0
Oct 30 03:30:17.342594 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.342114    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"
Oct 30 03:30:17.342594 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.342144    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" event={"ID":"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f","Type":"ContainerDied","Data":"c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426"}
Oct 30 03:30:17.342594 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.342201    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4" event={"ID":"d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f","Type":"ContainerDied","Data":"8ee9161a6c4ccfa81462b2de498292690165ac8a0a273d872350aacbee8d9755"}
Oct 30 03:30:17.342594 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.342222    5061 scope.go:122] "RemoveContainer" containerID="c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426"
Oct 30 03:30:17.346369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.346344    5061 generic.go:356] "Generic (PLEG): container finished" podID="70a5c3d4-194d-4214-89bc-697164722e46" containerID="165493583fded6369a0314b12c9ce1df39515edeef125461b8b497e04a5dda10" exitCode=60
Oct 30 03:30:17.346495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.346408    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd" event={"ID":"70a5c3d4-194d-4214-89bc-697164722e46","Type":"ContainerDied","Data":"165493583fded6369a0314b12c9ce1df39515edeef125461b8b497e04a5dda10"}
Oct 30 03:30:17.346709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.346693    5061 scope.go:122] "RemoveContainer" containerID="165493583fded6369a0314b12c9ce1df39515edeef125461b8b497e04a5dda10"
Oct 30 03:30:17.346936 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:17.346918    5061 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888850-tx2rd_odh-ai-gateway-infra(70a5c3d4-194d-4214-89bc-697164722e46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd" podUID="70a5c3d4-194d-4214-89bc-697164722e46"
Oct 30 03:30:17.360601 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.360580    5061 scope.go:122] "RemoveContainer" containerID="c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426"
Oct 30 03:30:17.361027 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:17.361003    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426\": container with ID starting with c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426 not found: ID does not exist" containerID="c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426"
Oct 30 03:30:17.361094 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.361037    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426"} err="failed to get container status \"c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426\": rpc error: code = NotFound desc = could not find container \"c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426\": container with ID starting with c6ea82bd29e1c8fe921b567035f765c8d0e9902adac566b11799e13434356426 not found: ID does not exist"
Oct 30 03:30:17.361094 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.361057    5061 scope.go:122] "RemoveContainer" containerID="d4d187214043c499a72af789028471e0fbc4c1696db6268a35b31406de9abcdb"
Oct 30 03:30:17.407941 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.407910    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"]
Oct 30 03:30:17.421712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.421687    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ae1dd9-openshift-default-668c46fcff-9qth4"]
Oct 30 03:30:17.421880 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.421727    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"]
Oct 30 03:30:17.514470 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.514443    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"
Oct 30 03:30:17.623675 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.623601    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-kube-api-access-qb448\" (UniqueName: \"kubernetes.io/projected/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-kube-api-access-qb448\") pod \"0b27eb1a-516b-4ec1-b5b4-afc8333f6984\" (UID: \"0b27eb1a-516b-4ec1-b5b4-afc8333f6984\") "
Oct 30 03:30:17.623812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.623792    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-maas-api-service-ca\") pod \"0b27eb1a-516b-4ec1-b5b4-afc8333f6984\" (UID: \"0b27eb1a-516b-4ec1-b5b4-afc8333f6984\") "
Oct 30 03:30:17.624242 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.624214    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-maas-api-service-ca" pod "0b27eb1a-516b-4ec1-b5b4-afc8333f6984" (UID: "0b27eb1a-516b-4ec1-b5b4-afc8333f6984"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:17.624391 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.624344    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-maas-api-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:17.625828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.625805    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-kube-api-access-qb448" pod "0b27eb1a-516b-4ec1-b5b4-afc8333f6984" (UID: "0b27eb1a-516b-4ec1-b5b4-afc8333f6984"). InnerVolumeSpecName "kube-api-access-qb448". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:17.727900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.726865    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb448\" (UniqueName: \"kubernetes.io/projected/0b27eb1a-516b-4ec1-b5b4-afc8333f6984-kube-api-access-qb448\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:17.845013 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:17.844977    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:18.352431 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.352399    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"
Oct 30 03:30:18.352944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.352398    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj" event={"ID":"0b27eb1a-516b-4ec1-b5b4-afc8333f6984","Type":"ContainerDied","Data":"be4161a6eaf1e8e6347989e1b0e4e851936db12e0ca7678f57c53f29786b56ba"}
Oct 30 03:30:18.352944 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.352553    5061 scope.go:122] "RemoveContainer" containerID="4ae14b969cdfe79922bc8123ac6c618e4534b5c0ba84b13035fbd46737602ea4"
Oct 30 03:30:18.432714 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.432682    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"]
Oct 30 03:30:18.435367 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.435323    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d823d1-29888850-nf9zj"]
Oct 30 03:30:18.506123 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.506096    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"
Oct 30 03:30:18.535777 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.535751    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a5c3d4-194d-4214-89bc-697164722e46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a5c3d4-194d-4214-89bc-697164722e46-maas-api-service-ca\") pod \"70a5c3d4-194d-4214-89bc-697164722e46\" (UID: \"70a5c3d4-194d-4214-89bc-697164722e46\") "
Oct 30 03:30:18.535952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.535913    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a5c3d4-194d-4214-89bc-697164722e46-kube-api-access-rf9rm\" (UniqueName: \"kubernetes.io/projected/70a5c3d4-194d-4214-89bc-697164722e46-kube-api-access-rf9rm\") pod \"70a5c3d4-194d-4214-89bc-697164722e46\" (UID: \"70a5c3d4-194d-4214-89bc-697164722e46\") "
Oct 30 03:30:18.536104 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.536081    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a5c3d4-194d-4214-89bc-697164722e46-maas-api-service-ca" pod "70a5c3d4-194d-4214-89bc-697164722e46" (UID: "70a5c3d4-194d-4214-89bc-697164722e46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:18.536258 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.536243    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a5c3d4-194d-4214-89bc-697164722e46-maas-api-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:18.537910 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.537890    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a5c3d4-194d-4214-89bc-697164722e46-kube-api-access-rf9rm" pod "70a5c3d4-194d-4214-89bc-697164722e46" (UID: "70a5c3d4-194d-4214-89bc-697164722e46"). InnerVolumeSpecName "kube-api-access-rf9rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:18.637636 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.637556    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9rm\" (UniqueName: \"kubernetes.io/projected/70a5c3d4-194d-4214-89bc-697164722e46-kube-api-access-rf9rm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:18.846161 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.846128    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" path="/var/lib/kubelet/pods/0b27eb1a-516b-4ec1-b5b4-afc8333f6984/volumes"
Oct 30 03:30:18.846724 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:18.846700    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" path="/var/lib/kubelet/pods/d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f/volumes"
Oct 30 03:30:19.296795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:19.296753    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:19.363180 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:19.363145    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"
Oct 30 03:30:19.363595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:19.363218    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd" event={"ID":"70a5c3d4-194d-4214-89bc-697164722e46","Type":"ContainerDied","Data":"cb66cbe0affa59b5c9839aaf6a69a3e8c39e9cbdff33f1fa55fb4aebbc199903"}
Oct 30 03:30:19.363595 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:19.363270    5061 scope.go:122] "RemoveContainer" containerID="165493583fded6369a0314b12c9ce1df39515edeef125461b8b497e04a5dda10"
Oct 30 03:30:19.408540 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:19.408511    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"]
Oct 30 03:30:19.412588 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:19.412566    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888850-tx2rd"]
Oct 30 03:30:20.849201 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:20.849168    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a5c3d4-194d-4214-89bc-697164722e46" path="/var/lib/kubelet/pods/70a5c3d4-194d-4214-89bc-697164722e46/volumes"
Oct 30 03:30:20.937185 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:20.937155    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"]
Oct 30 03:30:20.946054 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:20.946028    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"]
Oct 30 03:30:20.949644 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:20.949618    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"]
Oct 30 03:30:20.959521 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:20.959497    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:21.091831 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:21.091796    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:22.240654 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.240620    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm" podUID="d7a6eb60-0637-4d7c-9ab3-769ceb0cc464" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:30:22.240654 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.240639    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9" podUID="d29cfa43-cc24-4c84-a55d-12401012d345" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:30:22.497192 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.497109    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:22.785550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.785521    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:30:22.815784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.815756    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:30:22.881359 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.881298    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-kube-api-access-hfw7m\" (UniqueName: \"kubernetes.io/projected/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-kube-api-access-hfw7m\") pod \"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464\" (UID: \"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464\") "
Oct 30 03:30:22.881702 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.881399    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-plugins-config-volume\") pod \"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464\" (UID: \"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464\") "
Oct 30 03:30:22.881702 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.881495    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d29cfa43-cc24-4c84-a55d-12401012d345-kube-api-access-776x5\" (UniqueName: \"kubernetes.io/projected/d29cfa43-cc24-4c84-a55d-12401012d345-kube-api-access-776x5\") pod \"d29cfa43-cc24-4c84-a55d-12401012d345\" (UID: \"d29cfa43-cc24-4c84-a55d-12401012d345\") "
Oct 30 03:30:22.881702 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.881597    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d29cfa43-cc24-4c84-a55d-12401012d345-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d29cfa43-cc24-4c84-a55d-12401012d345-plugins-config-volume\") pod \"d29cfa43-cc24-4c84-a55d-12401012d345\" (UID: \"d29cfa43-cc24-4c84-a55d-12401012d345\") "
Oct 30 03:30:22.882125 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.882093    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-plugins-config-volume" pod "d7a6eb60-0637-4d7c-9ab3-769ceb0cc464" (UID: "d7a6eb60-0637-4d7c-9ab3-769ceb0cc464"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:22.882292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.882234    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29cfa43-cc24-4c84-a55d-12401012d345-plugins-config-volume" pod "d29cfa43-cc24-4c84-a55d-12401012d345" (UID: "d29cfa43-cc24-4c84-a55d-12401012d345"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:22.882733 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.882663    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.882733 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.882697    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d29cfa43-cc24-4c84-a55d-12401012d345-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.884662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.884634    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-kube-api-access-hfw7m" pod "d7a6eb60-0637-4d7c-9ab3-769ceb0cc464" (UID: "d7a6eb60-0637-4d7c-9ab3-769ceb0cc464"). InnerVolumeSpecName "kube-api-access-hfw7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:22.884842 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.884821    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29cfa43-cc24-4c84-a55d-12401012d345-kube-api-access-776x5" pod "d29cfa43-cc24-4c84-a55d-12401012d345" (UID: "d29cfa43-cc24-4c84-a55d-12401012d345"). InnerVolumeSpecName "kube-api-access-776x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:22.983697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.983654    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfw7m\" (UniqueName: \"kubernetes.io/projected/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464-kube-api-access-hfw7m\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.983697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:22.983690    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-776x5\" (UniqueName: \"kubernetes.io/projected/d29cfa43-cc24-4c84-a55d-12401012d345-kube-api-access-776x5\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:23.386607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.386569    5061 generic.go:356] "Generic (PLEG): container finished" podID="d7a6eb60-0637-4d7c-9ab3-769ceb0cc464" containerID="c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582" exitCode=0
Oct 30 03:30:23.387090 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.386622    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"
Oct 30 03:30:23.387090 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.386660    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm" event={"ID":"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464","Type":"ContainerDied","Data":"c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582"}
Oct 30 03:30:23.387090 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.386718    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm" event={"ID":"d7a6eb60-0637-4d7c-9ab3-769ceb0cc464","Type":"ContainerDied","Data":"e487eb0c09e7637d301c0c547c82a645c437e3f7411a3a3a6300dda811cb5d86"}
Oct 30 03:30:23.387090 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.386740    5061 scope.go:122] "RemoveContainer" containerID="c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582"
Oct 30 03:30:23.388279 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.388249    5061 generic.go:356] "Generic (PLEG): container finished" podID="d29cfa43-cc24-4c84-a55d-12401012d345" containerID="1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4" exitCode=0
Oct 30 03:30:23.388413 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.388281    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9" event={"ID":"d29cfa43-cc24-4c84-a55d-12401012d345","Type":"ContainerDied","Data":"1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4"}
Oct 30 03:30:23.388413 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.388299    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"
Oct 30 03:30:23.388413 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.388313    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9" event={"ID":"d29cfa43-cc24-4c84-a55d-12401012d345","Type":"ContainerDied","Data":"616ee5a30a24dce5012443ff1d2ef1042a271836c0bf0649b327f7eadea0ceb4"}
Oct 30 03:30:23.397767 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.397748    5061 scope.go:122] "RemoveContainer" containerID="c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582"
Oct 30 03:30:23.398128 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:23.398095    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582\": container with ID starting with c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582 not found: ID does not exist" containerID="c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582"
Oct 30 03:30:23.398229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.398133    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582"} err="failed to get container status \"c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582\": rpc error: code = NotFound desc = could not find container \"c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582\": container with ID starting with c4b965a5025c369e5c1b82b63d185c7dff50cfe3729511ae5d6fac4154245582 not found: ID does not exist"
Oct 30 03:30:23.398229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.398149    5061 scope.go:122] "RemoveContainer" containerID="1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4"
Oct 30 03:30:23.406797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.406775    5061 scope.go:122] "RemoveContainer" containerID="1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4"
Oct 30 03:30:23.407087 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:23.407069    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4\": container with ID starting with 1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4 not found: ID does not exist" containerID="1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4"
Oct 30 03:30:23.407157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.407094    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4"} err="failed to get container status \"1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4\": rpc error: code = NotFound desc = could not find container \"1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4\": container with ID starting with 1e1cdb3beb219827854965b8f4e7c24308533e0a3473e1cddb13509e6b071ad4 not found: ID does not exist"
Oct 30 03:30:23.432480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.432449    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"]
Oct 30 03:30:23.442940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.442916    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ae1dd9-5b9d84b87b-24ftm"]
Oct 30 03:30:23.468168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.468140    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"]
Oct 30 03:30:23.472068 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:23.472046    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ae1dd9-55bb58fb95-c82n9"]
Oct 30 03:30:24.112522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:24.112490    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:24.847082 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:24.847052    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d29cfa43-cc24-4c84-a55d-12401012d345" path="/var/lib/kubelet/pods/d29cfa43-cc24-4c84-a55d-12401012d345/volumes"
Oct 30 03:30:24.847478 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:24.847418    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a6eb60-0637-4d7c-9ab3-769ceb0cc464" path="/var/lib/kubelet/pods/d7a6eb60-0637-4d7c-9ab3-769ceb0cc464/volumes"
Oct 30 03:30:25.446541 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:25.446501    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:30:26.150599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:26.150565    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:26.747750 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:26.747722    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:26.853472 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:26.853436    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:27.664799 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:27.664757    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:29.285139 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:29.285089    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:30.641914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:30.641876    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:30.702135 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:30.702102    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"]
Oct 30 03:30:30.703149 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:30.702317    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8" podUID="338661f3-a26f-486a-a63e-e7b10c7499e2" containerName="payload-processing" containerID="cri-o://14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b" gracePeriod=30
Oct 30 03:30:30.729184 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:30.729150    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"]
Oct 30 03:30:30.729405 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:30.729362    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf" podUID="bb0624f0-1f31-469a-84b5-97b1a9c42a8e" containerName="payload-pre-processing" containerID="cri-o://5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a" gracePeriod=30
Oct 30 03:30:30.800258 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:30.800232    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:30.919967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:30.919923    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:30.999012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:30.998980    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:30:31.030692 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.030666    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:30:31.076983 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.075279    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338661f3-a26f-486a-a63e-e7b10c7499e2-kube-api-access-kbp4w\" (UniqueName: \"kubernetes.io/projected/338661f3-a26f-486a-a63e-e7b10c7499e2-kube-api-access-kbp4w\") pod \"338661f3-a26f-486a-a63e-e7b10c7499e2\" (UID: \"338661f3-a26f-486a-a63e-e7b10c7499e2\") "
Oct 30 03:30:31.076983 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.076070    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/338661f3-a26f-486a-a63e-e7b10c7499e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338661f3-a26f-486a-a63e-e7b10c7499e2-plugins-config-volume\") pod \"338661f3-a26f-486a-a63e-e7b10c7499e2\" (UID: \"338661f3-a26f-486a-a63e-e7b10c7499e2\") "
Oct 30 03:30:31.078580 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.077656    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338661f3-a26f-486a-a63e-e7b10c7499e2-plugins-config-volume" pod "338661f3-a26f-486a-a63e-e7b10c7499e2" (UID: "338661f3-a26f-486a-a63e-e7b10c7499e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:31.084292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.083875    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338661f3-a26f-486a-a63e-e7b10c7499e2-kube-api-access-kbp4w" pod "338661f3-a26f-486a-a63e-e7b10c7499e2" (UID: "338661f3-a26f-486a-a63e-e7b10c7499e2"). InnerVolumeSpecName "kube-api-access-kbp4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:31.164720 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.164643    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:31.177870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.177831    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-kube-api-access-dkqq9\" (UniqueName: \"kubernetes.io/projected/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-kube-api-access-dkqq9\") pod \"bb0624f0-1f31-469a-84b5-97b1a9c42a8e\" (UID: \"bb0624f0-1f31-469a-84b5-97b1a9c42a8e\") "
Oct 30 03:30:31.178036 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.178021    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-plugins-config-volume\") pod \"bb0624f0-1f31-469a-84b5-97b1a9c42a8e\" (UID: \"bb0624f0-1f31-469a-84b5-97b1a9c42a8e\") "
Oct 30 03:30:31.178304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.178274    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338661f3-a26f-486a-a63e-e7b10c7499e2-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:31.178304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.178291    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbp4w\" (UniqueName: \"kubernetes.io/projected/338661f3-a26f-486a-a63e-e7b10c7499e2-kube-api-access-kbp4w\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:31.178466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.178355    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-plugins-config-volume" pod "bb0624f0-1f31-469a-84b5-97b1a9c42a8e" (UID: "bb0624f0-1f31-469a-84b5-97b1a9c42a8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:31.179912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.179891    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-kube-api-access-dkqq9" pod "bb0624f0-1f31-469a-84b5-97b1a9c42a8e" (UID: "bb0624f0-1f31-469a-84b5-97b1a9c42a8e"). InnerVolumeSpecName "kube-api-access-dkqq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:31.268972 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.268932    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"]
Oct 30 03:30:31.274020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.273991    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf605a-q87qg"]
Oct 30 03:30:31.279370 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.279343    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkqq9\" (UniqueName: \"kubernetes.io/projected/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-kube-api-access-dkqq9\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:31.279370 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.279370    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0624f0-1f31-469a-84b5-97b1a9c42a8e-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:31.280994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.280968    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:31.401678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.401638    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:31.434172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.434086    5061 generic.go:356] "Generic (PLEG): container finished" podID="338661f3-a26f-486a-a63e-e7b10c7499e2" containerID="14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b" exitCode=0
Oct 30 03:30:31.434172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.434130    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"
Oct 30 03:30:31.434404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.434177    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8" event={"ID":"338661f3-a26f-486a-a63e-e7b10c7499e2","Type":"ContainerDied","Data":"14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b"}
Oct 30 03:30:31.434404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.434238    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8" event={"ID":"338661f3-a26f-486a-a63e-e7b10c7499e2","Type":"ContainerDied","Data":"a3a337fe451f33774f76be8dc4493e45b9a0c20d45bb5c3f9176debc9a763adf"}
Oct 30 03:30:31.434404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.434269    5061 scope.go:122] "RemoveContainer" containerID="14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b"
Oct 30 03:30:31.435693 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.435672    5061 generic.go:356] "Generic (PLEG): container finished" podID="bb0624f0-1f31-469a-84b5-97b1a9c42a8e" containerID="5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a" exitCode=0
Oct 30 03:30:31.435814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.435724    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"
Oct 30 03:30:31.435814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.435753    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf" event={"ID":"bb0624f0-1f31-469a-84b5-97b1a9c42a8e","Type":"ContainerDied","Data":"5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a"}
Oct 30 03:30:31.435814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.435790    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf" event={"ID":"bb0624f0-1f31-469a-84b5-97b1a9c42a8e","Type":"ContainerDied","Data":"4c5f4ade1893d20e2f21e2fcdb59fd006dde6cd28e5433d72fc7a0253b0e0beb"}
Oct 30 03:30:31.445038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.445021    5061 scope.go:122] "RemoveContainer" containerID="14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b"
Oct 30 03:30:31.445364 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:31.445342    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b\": container with ID starting with 14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b not found: ID does not exist" containerID="14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b"
Oct 30 03:30:31.445449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.445377    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b"} err="failed to get container status \"14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b\": rpc error: code = NotFound desc = could not find container \"14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b\": container with ID starting with 14723d1a10a6f527f2acb87a6f650c6c8ab672944f8782fe05bf4f54ac6b805b not found: ID does not exist"
Oct 30 03:30:31.445449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.445399    5061 scope.go:122] "RemoveContainer" containerID="5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a"
Oct 30 03:30:31.454415 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.454396    5061 scope.go:122] "RemoveContainer" containerID="5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a"
Oct 30 03:30:31.454659 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:31.454641    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a\": container with ID starting with 5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a not found: ID does not exist" containerID="5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a"
Oct 30 03:30:31.454705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.454668    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a"} err="failed to get container status \"5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a\": rpc error: code = NotFound desc = could not find container \"5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a\": container with ID starting with 5b61a25261ddaf4386fb2b730ea2acbaea3a87cb4906e5ad8c62fd23d9d6137a not found: ID does not exist"
Oct 30 03:30:31.491649 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.491605    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"]
Oct 30 03:30:31.495643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.495616    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf605a-95c4f479f-hmpdf"]
Oct 30 03:30:31.525287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.525253    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"]
Oct 30 03:30:31.529176 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.529154    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf605a-788659d948-6j8m8"]
Oct 30 03:30:31.792943 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:31.792907    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:32.853186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:32.853095    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="338661f3-a26f-486a-a63e-e7b10c7499e2" path="/var/lib/kubelet/pods/338661f3-a26f-486a-a63e-e7b10c7499e2/volumes"
Oct 30 03:30:32.854386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:32.854351    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ce2051-8166-4b9c-94f2-2711be84b2d1" path="/var/lib/kubelet/pods/62ce2051-8166-4b9c-94f2-2711be84b2d1/volumes"
Oct 30 03:30:32.855279 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:32.855239    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0624f0-1f31-469a-84b5-97b1a9c42a8e" path="/var/lib/kubelet/pods/bb0624f0-1f31-469a-84b5-97b1a9c42a8e/volumes"
Oct 30 03:30:32.858558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:32.858483    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:35.002591 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:35.002511    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:35.958498 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:35.958451    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" containerID="cri-o://523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93" gracePeriod=45
Oct 30 03:30:35.967151 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:35.967112    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main" containerID="cri-o://65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187" gracePeriod=45
Oct 30 03:30:35.971309 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:35.971277    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main" containerID="cri-o://ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924" gracePeriod=45
Oct 30 03:30:36.246054 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.246025    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 03:30:36.268770 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.268734    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 03:30:36.272419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.272396    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 03:30:36.363673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.363644    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c837855-e677-46af-91b9-2f2a06306d5b-kube-api-access-z2thm\" (UniqueName: \"kubernetes.io/projected/2c837855-e677-46af-91b9-2f2a06306d5b-kube-api-access-z2thm\") pod \"2c837855-e677-46af-91b9-2f2a06306d5b\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") "
Oct 30 03:30:36.363834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.363788    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-dshm\") pod \"2c837855-e677-46af-91b9-2f2a06306d5b\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") "
Oct 30 03:30:36.363926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.363834    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-home\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-home\") pod \"2c837855-e677-46af-91b9-2f2a06306d5b\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") "
Oct 30 03:30:36.363926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.363917    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-tmp-dir\") pod \"2c837855-e677-46af-91b9-2f2a06306d5b\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") "
Oct 30 03:30:36.364050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.363964    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c837855-e677-46af-91b9-2f2a06306d5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c837855-e677-46af-91b9-2f2a06306d5b-tls-certs\") pod \"2c837855-e677-46af-91b9-2f2a06306d5b\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") "
Oct 30 03:30:36.364050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.364027    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-model-cache\") pod \"2c837855-e677-46af-91b9-2f2a06306d5b\" (UID: \"2c837855-e677-46af-91b9-2f2a06306d5b\") "
Oct 30 03:30:36.364570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.364313    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-tmp-dir" pod "2c837855-e677-46af-91b9-2f2a06306d5b" (UID: "2c837855-e677-46af-91b9-2f2a06306d5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.364570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.364454    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.364570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.364524    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-model-cache" pod "2c837855-e677-46af-91b9-2f2a06306d5b" (UID: "2c837855-e677-46af-91b9-2f2a06306d5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.364570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.364541    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-home" pod "2c837855-e677-46af-91b9-2f2a06306d5b" (UID: "2c837855-e677-46af-91b9-2f2a06306d5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.366731 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.366625    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-dshm" pod "2c837855-e677-46af-91b9-2f2a06306d5b" (UID: "2c837855-e677-46af-91b9-2f2a06306d5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.366731 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.366687    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c837855-e677-46af-91b9-2f2a06306d5b-tls-certs" pod "2c837855-e677-46af-91b9-2f2a06306d5b" (UID: "2c837855-e677-46af-91b9-2f2a06306d5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:36.366977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.366950    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c837855-e677-46af-91b9-2f2a06306d5b-kube-api-access-z2thm" pod "2c837855-e677-46af-91b9-2f2a06306d5b" (UID: "2c837855-e677-46af-91b9-2f2a06306d5b"). InnerVolumeSpecName "kube-api-access-z2thm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:36.465910 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.465872    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-dshm\") pod \"ec889f00-32b6-4c2c-931e-f42c26a0026c\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") "
Oct 30 03:30:36.466096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.465924    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-home\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-home\") pod \"ec889f00-32b6-4c2c-931e-f42c26a0026c\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") "
Oct 30 03:30:36.466096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.465977    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec889f00-32b6-4c2c-931e-f42c26a0026c-kube-api-access-pcm6l\" (UniqueName: \"kubernetes.io/projected/ec889f00-32b6-4c2c-931e-f42c26a0026c-kube-api-access-pcm6l\") pod \"ec889f00-32b6-4c2c-931e-f42c26a0026c\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") "
Oct 30 03:30:36.466096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466006    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tls-certs\" (UniqueName: \"kubernetes.io/secret/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tls-certs\") pod \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") "
Oct 30 03:30:36.466096 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466037    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec889f00-32b6-4c2c-931e-f42c26a0026c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec889f00-32b6-4c2c-931e-f42c26a0026c-tls-certs\") pod \"ec889f00-32b6-4c2c-931e-f42c26a0026c\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") "
Oct 30 03:30:36.466338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466142    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-model-cache\") pod \"ec889f00-32b6-4c2c-931e-f42c26a0026c\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") "
Oct 30 03:30:36.466338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466164    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-home" pod "ec889f00-32b6-4c2c-931e-f42c26a0026c" (UID: "ec889f00-32b6-4c2c-931e-f42c26a0026c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.466338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466247    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-kube-api-access-czzkp\" (UniqueName: \"kubernetes.io/projected/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-kube-api-access-czzkp\") pod \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") "
Oct 30 03:30:36.466338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466299    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-home\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-home\") pod \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") "
Oct 30 03:30:36.466542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466347    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-tmp-dir\") pod \"ec889f00-32b6-4c2c-931e-f42c26a0026c\" (UID: \"ec889f00-32b6-4c2c-931e-f42c26a0026c\") "
Oct 30 03:30:36.466542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466376    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-model-cache" pod "ec889f00-32b6-4c2c-931e-f42c26a0026c" (UID: "ec889f00-32b6-4c2c-931e-f42c26a0026c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.466542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466428    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-dshm\") pod \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") "
Oct 30 03:30:36.466542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466463    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tmp-dir\") pod \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") "
Oct 30 03:30:36.466542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466500    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-model-cache\") pod \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\" (UID: \"ede5a7fc-13b5-40ea-a716-7b0acdaacc93\") "
Oct 30 03:30:36.466824 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466576    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-home" pod "ede5a7fc-13b5-40ea-a716-7b0acdaacc93" (UID: "ede5a7fc-13b5-40ea-a716-7b0acdaacc93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.466908 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.466872    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tmp-dir" pod "ede5a7fc-13b5-40ea-a716-7b0acdaacc93" (UID: "ede5a7fc-13b5-40ea-a716-7b0acdaacc93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.467098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.467059    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-tmp-dir" pod "ec889f00-32b6-4c2c-931e-f42c26a0026c" (UID: "ec889f00-32b6-4c2c-931e-f42c26a0026c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.467098 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.467080    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-model-cache" pod "ede5a7fc-13b5-40ea-a716-7b0acdaacc93" (UID: "ede5a7fc-13b5-40ea-a716-7b0acdaacc93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.467248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.467143    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.467248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.467162    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2thm\" (UniqueName: \"kubernetes.io/projected/2c837855-e677-46af-91b9-2f2a06306d5b-kube-api-access-z2thm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.467248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.467176    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.467248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.467190    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.467248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.467203    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.467248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.467215    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.467248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.467227    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.467248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.467240    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c837855-e677-46af-91b9-2f2a06306d5b-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.467513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.467252    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c837855-e677-46af-91b9-2f2a06306d5b-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.468526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.468494    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-dshm" pod "ede5a7fc-13b5-40ea-a716-7b0acdaacc93" (UID: "ede5a7fc-13b5-40ea-a716-7b0acdaacc93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.468790 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.468762    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-dshm" pod "ec889f00-32b6-4c2c-931e-f42c26a0026c" (UID: "ec889f00-32b6-4c2c-931e-f42c26a0026c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:36.468876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.468776    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tls-certs" pod "ede5a7fc-13b5-40ea-a716-7b0acdaacc93" (UID: "ede5a7fc-13b5-40ea-a716-7b0acdaacc93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:36.468876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.468796    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec889f00-32b6-4c2c-931e-f42c26a0026c-kube-api-access-pcm6l" pod "ec889f00-32b6-4c2c-931e-f42c26a0026c" (UID: "ec889f00-32b6-4c2c-931e-f42c26a0026c"). InnerVolumeSpecName "kube-api-access-pcm6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:36.468996 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.468975    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec889f00-32b6-4c2c-931e-f42c26a0026c-tls-certs" pod "ec889f00-32b6-4c2c-931e-f42c26a0026c" (UID: "ec889f00-32b6-4c2c-931e-f42c26a0026c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:36.469275 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.469257    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-kube-api-access-czzkp" pod "ede5a7fc-13b5-40ea-a716-7b0acdaacc93" (UID: "ede5a7fc-13b5-40ea-a716-7b0acdaacc93"). InnerVolumeSpecName "kube-api-access-czzkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:36.482634 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.482581    5061 generic.go:356] "Generic (PLEG): container finished" podID="2c837855-e677-46af-91b9-2f2a06306d5b" containerID="523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93" exitCode=0
Oct 30 03:30:36.482634 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.482624    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"
Oct 30 03:30:36.482773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.482659    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" event={"ID":"2c837855-e677-46af-91b9-2f2a06306d5b","Type":"ContainerDied","Data":"523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93"}
Oct 30 03:30:36.482773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.482701    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" event={"ID":"2c837855-e677-46af-91b9-2f2a06306d5b","Type":"ContainerDied","Data":"5010bdf4c6f05458c898df586e39cb6218414f4f85ede8be4c3fdc1398701bd0"}
Oct 30 03:30:36.482773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.482718    5061 scope.go:122] "RemoveContainer" containerID="523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93"
Oct 30 03:30:36.484124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.484081    5061 generic.go:356] "Generic (PLEG): container finished" podID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerID="ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924" exitCode=0
Oct 30 03:30:36.484124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.484111    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" event={"ID":"ede5a7fc-13b5-40ea-a716-7b0acdaacc93","Type":"ContainerDied","Data":"ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924"}
Oct 30 03:30:36.484301 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.484126    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"
Oct 30 03:30:36.484301 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.484143    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs" event={"ID":"ede5a7fc-13b5-40ea-a716-7b0acdaacc93","Type":"ContainerDied","Data":"3fd2df603924cac52bf5a3587184f88078163d1c19cb2bd1a610796fdf7457d1"}
Oct 30 03:30:36.485726 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.485699    5061 generic.go:356] "Generic (PLEG): container finished" podID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerID="65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187" exitCode=0
Oct 30 03:30:36.485904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.485743    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"
Oct 30 03:30:36.485904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.485776    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" event={"ID":"ec889f00-32b6-4c2c-931e-f42c26a0026c","Type":"ContainerDied","Data":"65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187"}
Oct 30 03:30:36.485904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.485817    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd" event={"ID":"ec889f00-32b6-4c2c-931e-f42c26a0026c","Type":"ContainerDied","Data":"ad5560e2eeb7bd8574b71ee5f2ad2ae13ec38545fdb5bf22fdad8a9de788cccd"}
Oct 30 03:30:36.487833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.487808    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:36.495678 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.495231    5061 scope.go:122] "RemoveContainer" containerID="523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93"
Oct 30 03:30:36.495678 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:36.495630    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93\": container with ID starting with 523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93 not found: ID does not exist" containerID="523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93"
Oct 30 03:30:36.495904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.495667    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93"} err="failed to get container status \"523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93\": rpc error: code = NotFound desc = could not find container \"523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93\": container with ID starting with 523a45af140b925700a5b14abf2d00350d25d5010dea730a9f42da1a7e301e93 not found: ID does not exist"
Oct 30 03:30:36.495904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.495696    5061 scope.go:122] "RemoveContainer" containerID="ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924"
Oct 30 03:30:36.510036 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.510018    5061 scope.go:122] "RemoveContainer" containerID="ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924"
Oct 30 03:30:36.510313 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:36.510291    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924\": container with ID starting with ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924 not found: ID does not exist" containerID="ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924"
Oct 30 03:30:36.510389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.510324    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924"} err="failed to get container status \"ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924\": rpc error: code = NotFound desc = could not find container \"ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924\": container with ID starting with ddd042254e8f05f9e04937db9e650dff249379bb3bd837d0e656d2c75be09924 not found: ID does not exist"
Oct 30 03:30:36.510389 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.510348    5061 scope.go:122] "RemoveContainer" containerID="65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187"
Oct 30 03:30:36.518059 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.518042    5061 scope.go:122] "RemoveContainer" containerID="65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187"
Oct 30 03:30:36.518325 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:36.518310    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187\": container with ID starting with 65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187 not found: ID does not exist" containerID="65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187"
Oct 30 03:30:36.518375 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.518338    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187"} err="failed to get container status \"65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187\": rpc error: code = NotFound desc = could not find container \"65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187\": container with ID starting with 65298d12cc6342c75009dd07492770f3d365e840b952814a7b4c222a35201187 not found: ID does not exist"
Oct 30 03:30:36.555115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.555080    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"]
Oct 30 03:30:36.557223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.557194    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-w5-f7d1a1-kserve-6f8fb9ff9d-z5zvd"]
Oct 30 03:30:36.568041 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.568016    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czzkp\" (UniqueName: \"kubernetes.io/projected/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-kube-api-access-czzkp\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.568182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.568045    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.568182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.568062    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.568182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.568075    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.568182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.568086    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec889f00-32b6-4c2c-931e-f42c26a0026c-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.568182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.568145    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcm6l\" (UniqueName: \"kubernetes.io/projected/ec889f00-32b6-4c2c-931e-f42c26a0026c-kube-api-access-pcm6l\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.568182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.568158    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede5a7fc-13b5-40ea-a716-7b0acdaacc93-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.568182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.568171    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec889f00-32b6-4c2c-931e-f42c26a0026c-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:30:36.586357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.586332    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"]
Oct 30 03:30:36.591321 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.591296    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/premium-w5-f7d1a1-kserve-7dfc9448bc-fwtcs"]
Oct 30 03:30:36.620457 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.620432    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"]
Oct 30 03:30:36.623939 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.623917    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn"]
Oct 30 03:30:36.819396 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.819363    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:36.850980 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.850932    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" path="/var/lib/kubelet/pods/2c837855-e677-46af-91b9-2f2a06306d5b/volumes"
Oct 30 03:30:36.851539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.851503    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" path="/var/lib/kubelet/pods/ec889f00-32b6-4c2c-931e-f42c26a0026c/volumes"
Oct 30 03:30:36.852018 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:36.851999    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" path="/var/lib/kubelet/pods/ede5a7fc-13b5-40ea-a716-7b0acdaacc93/volumes"
Oct 30 03:30:37.202869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:37.202756    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f7d1a1/distinct-2-w5-f7d1a1-kserve-596bddd4bf-cm6zn" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 03:30:38.089178 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:38.089141    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:39.681625 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:39.681594    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:41.275248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:41.275216    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:43.294603 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:43.294567    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:44.818109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:44.818077    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:46.469256 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:46.469222    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:48.034469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:48.034433    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:49.552587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:49.552547    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:50.785203 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:50.785162    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:51.787233 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:51.787200    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:53.120709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:53.120672    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:53.566721 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:53.566689    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:53.686145 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:53.686115    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:54.195582 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:54.195538    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:55.247784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:55.247748    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:55.450255 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:30:55.450218    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:30:55.460815 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:55.460791    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:30:55.463753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:55.463728    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:30:55.468156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:55.468136    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:30:55.470918 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:55.470894    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:30:56.916083 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:56.916042    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:58.591641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:58.591608    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:58.833058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:58.833023    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:59.076381 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:59.076332    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:59.380265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:59.380179    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:59.527029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:30:59.526993    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:00.016217 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:00.016184    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:00.500205 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:00.500172    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:00.624419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:00.624389    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:00.768122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:00.768039    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:00.782440 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:00.782404    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"]
Oct 30 03:31:00.782691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:00.782661    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy" containerID="cri-o://ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c" gracePeriod=30
Oct 30 03:31:00.890500 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:00.890460    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:02.267455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:02.267422    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:03.980394 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:03.980357    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:05.683436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:05.683357    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:06.031267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.031239    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 03:31:06.095587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.095555    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-socket\") pod \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") "
Oct 30 03:31:06.095731 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.095610    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-credential-socket\") pod \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") "
Oct 30 03:31:06.095731 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.095655    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-certs\") pod \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") "
Oct 30 03:31:06.095731 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.095711    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istiod-ca-cert\") pod \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") "
Oct 30 03:31:06.095935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.095749    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-data\") pod \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") "
Oct 30 03:31:06.095935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.095779    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-podinfo\") pod \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") "
Oct 30 03:31:06.095935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.095830    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-socket" pod "c1227d4b-32b5-4ec8-9037-0b830b4528c6" (UID: "c1227d4b-32b5-4ec8-9037-0b830b4528c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:06.095935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.095870    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-kube-api-access-jgnpr\" (UniqueName: \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-kube-api-access-jgnpr\") pod \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") "
Oct 30 03:31:06.095935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.095933    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-envoy\") pod \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") "
Oct 30 03:31:06.096202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.095962    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-certs" pod "c1227d4b-32b5-4ec8-9037-0b830b4528c6" (UID: "c1227d4b-32b5-4ec8-9037-0b830b4528c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:06.096202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.095986    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-credential-socket" pod "c1227d4b-32b5-4ec8-9037-0b830b4528c6" (UID: "c1227d4b-32b5-4ec8-9037-0b830b4528c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:06.096202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.096082    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-data" pod "c1227d4b-32b5-4ec8-9037-0b830b4528c6" (UID: "c1227d4b-32b5-4ec8-9037-0b830b4528c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:06.096202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.096105    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-token\" (UniqueName: \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-token\") pod \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\" (UID: \"c1227d4b-32b5-4ec8-9037-0b830b4528c6\") "
Oct 30 03:31:06.096587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.096526    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.096587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.096555    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.096587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.096558    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istiod-ca-cert" pod "c1227d4b-32b5-4ec8-9037-0b830b4528c6" (UID: "c1227d4b-32b5-4ec8-9037-0b830b4528c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:06.096587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.096570    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.096587 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.096583    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.098724 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.098696    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-envoy" pod "c1227d4b-32b5-4ec8-9037-0b830b4528c6" (UID: "c1227d4b-32b5-4ec8-9037-0b830b4528c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:06.098870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.098697    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-kube-api-access-jgnpr" pod "c1227d4b-32b5-4ec8-9037-0b830b4528c6" (UID: "c1227d4b-32b5-4ec8-9037-0b830b4528c6"). InnerVolumeSpecName "kube-api-access-jgnpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:06.098941 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.098898    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-podinfo" pod "c1227d4b-32b5-4ec8-9037-0b830b4528c6" (UID: "c1227d4b-32b5-4ec8-9037-0b830b4528c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:31:06.101955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.101932    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-token" pod "c1227d4b-32b5-4ec8-9037-0b830b4528c6" (UID: "c1227d4b-32b5-4ec8-9037-0b830b4528c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:06.107698 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.107656    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:06.197205 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.197178    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.197205 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.197207    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.197376 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.197220    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.197376 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.197233    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgnpr\" (UniqueName: \"kubernetes.io/projected/c1227d4b-32b5-4ec8-9037-0b830b4528c6-kube-api-access-jgnpr\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.197376 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.197246    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1227d4b-32b5-4ec8-9037-0b830b4528c6-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:06.649698 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.649656    5061 generic.go:356] "Generic (PLEG): container finished" podID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerID="ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c" exitCode=0
Oct 30 03:31:06.649898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.649701    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"
Oct 30 03:31:06.649898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.649716    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" event={"ID":"c1227d4b-32b5-4ec8-9037-0b830b4528c6","Type":"ContainerDied","Data":"ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c"}
Oct 30 03:31:06.649898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.649761    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn" event={"ID":"c1227d4b-32b5-4ec8-9037-0b830b4528c6","Type":"ContainerDied","Data":"4555d7f792d5e9d680b45fe25ff22acf10273cdaccbd129b7ae246b5d49abfe2"}
Oct 30 03:31:06.649898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.649782    5061 scope.go:122] "RemoveContainer" containerID="ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c"
Oct 30 03:31:06.660286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.660263    5061 scope.go:122] "RemoveContainer" containerID="ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c"
Oct 30 03:31:06.660555 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:31:06.660532    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c\": container with ID starting with ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c not found: ID does not exist" containerID="ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c"
Oct 30 03:31:06.660655 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.660562    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c"} err="failed to get container status \"ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c\": rpc error: code = NotFound desc = could not find container \"ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c\": container with ID starting with ddeb03b0d63d7c4156c629bfc117696f58a7e925fc629fd4382556b473f2e95c not found: ID does not exist"
Oct 30 03:31:06.705509 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.705485    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"]
Oct 30 03:31:06.715590 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.715566    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7d1a1-openshift-default-7fcf76dfc9-wfmmn"]
Oct 30 03:31:06.850697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:06.850666    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" path="/var/lib/kubelet/pods/c1227d4b-32b5-4ec8-9037-0b830b4528c6/volumes"
Oct 30 03:31:07.289957 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:07.289923    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:09.382191 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:09.382154    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:10.910662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:10.910629    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:12.446490 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:12.446458    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:13.578594 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:13.578558    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:14.174753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:14.174718    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:14.283194 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:14.283156    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:15.745994 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:15.745960    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:17.663164 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:17.663125    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:19.090866 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:19.090811    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:20.020216 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:20.020184    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:20.151983 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:20.151919    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:20.372814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:20.372774    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:20.785200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:20.785113    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:22.587050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:22.587015    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:24.126423 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:24.126386    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:25.445344 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:31:25.445310    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:31:26.233876 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:26.233822    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:27.703000 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:27.702965    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:29.386094 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:29.386059    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:30.839265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:30.839237    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:32.033235 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.033202    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"]
Oct 30 03:31:32.033883 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.033489    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h" podUID="c2fedcd9-ed61-4532-ae59-8583c533412e" containerName="maas-api" containerID="cri-o://3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76" gracePeriod=30
Oct 30 03:31:32.126447 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.126418    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:32.189768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.189736    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"]
Oct 30 03:31:32.189984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.189958    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk" podUID="e3a5a139-8b79-485f-9392-4fd1b0f18561" containerName="payload-processing" containerID="cri-o://6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00" gracePeriod=30
Oct 30 03:31:32.214474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.213290    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"]
Oct 30 03:31:32.214474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.213501    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565" podUID="b08d7a98-4183-4f36-918d-3a5df4aaa0ab" containerName="payload-pre-processing" containerID="cri-o://1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4" gracePeriod=30
Oct 30 03:31:32.299116 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.299077    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:32.348923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.348891    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:31:32.416575 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.416541    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:32.459735 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.459696    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fedcd9-ed61-4532-ae59-8583c533412e-kube-api-access-clqpl\" (UniqueName: \"kubernetes.io/projected/c2fedcd9-ed61-4532-ae59-8583c533412e-kube-api-access-clqpl\") pod \"c2fedcd9-ed61-4532-ae59-8583c533412e\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") "
Oct 30 03:31:32.459982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.459787    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2fedcd9-ed61-4532-ae59-8583c533412e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2fedcd9-ed61-4532-ae59-8583c533412e-openshift-service-ca\") pod \"c2fedcd9-ed61-4532-ae59-8583c533412e\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") "
Oct 30 03:31:32.459982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.459834    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-metrics-certs\") pod \"c2fedcd9-ed61-4532-ae59-8583c533412e\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") "
Oct 30 03:31:32.459982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.459955    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-maas-api-tls\") pod \"c2fedcd9-ed61-4532-ae59-8583c533412e\" (UID: \"c2fedcd9-ed61-4532-ae59-8583c533412e\") "
Oct 30 03:31:32.460386 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.460362    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fedcd9-ed61-4532-ae59-8583c533412e-openshift-service-ca" pod "c2fedcd9-ed61-4532-ae59-8583c533412e" (UID: "c2fedcd9-ed61-4532-ae59-8583c533412e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:32.462556 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.462526    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fedcd9-ed61-4532-ae59-8583c533412e-kube-api-access-clqpl" pod "c2fedcd9-ed61-4532-ae59-8583c533412e" (UID: "c2fedcd9-ed61-4532-ae59-8583c533412e"). InnerVolumeSpecName "kube-api-access-clqpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:32.462681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.462584    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-maas-api-tls" pod "c2fedcd9-ed61-4532-ae59-8583c533412e" (UID: "c2fedcd9-ed61-4532-ae59-8583c533412e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:32.462834 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.462809    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-metrics-certs" pod "c2fedcd9-ed61-4532-ae59-8583c533412e" (UID: "c2fedcd9-ed61-4532-ae59-8583c533412e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:32.464783 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.464753    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:31:32.497712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.497564    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:31:32.561290 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.561260    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a5a139-8b79-485f-9392-4fd1b0f18561-kube-api-access-6w55k\" (UniqueName: \"kubernetes.io/projected/e3a5a139-8b79-485f-9392-4fd1b0f18561-kube-api-access-6w55k\") pod \"e3a5a139-8b79-485f-9392-4fd1b0f18561\" (UID: \"e3a5a139-8b79-485f-9392-4fd1b0f18561\") "
Oct 30 03:31:32.561457 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.561411    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a5a139-8b79-485f-9392-4fd1b0f18561-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a5a139-8b79-485f-9392-4fd1b0f18561-plugins-config-volume\") pod \"e3a5a139-8b79-485f-9392-4fd1b0f18561\" (UID: \"e3a5a139-8b79-485f-9392-4fd1b0f18561\") "
Oct 30 03:31:32.561788 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.561642    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:32.561788 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.561657    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clqpl\" (UniqueName: \"kubernetes.io/projected/c2fedcd9-ed61-4532-ae59-8583c533412e-kube-api-access-clqpl\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:32.561788 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.561667    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2fedcd9-ed61-4532-ae59-8583c533412e-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:32.561788 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.561675    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2fedcd9-ed61-4532-ae59-8583c533412e-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:32.561788 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.561773    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a5a139-8b79-485f-9392-4fd1b0f18561-plugins-config-volume" pod "e3a5a139-8b79-485f-9392-4fd1b0f18561" (UID: "e3a5a139-8b79-485f-9392-4fd1b0f18561"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:32.563467 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.563447    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a5a139-8b79-485f-9392-4fd1b0f18561-kube-api-access-6w55k" pod "e3a5a139-8b79-485f-9392-4fd1b0f18561" (UID: "e3a5a139-8b79-485f-9392-4fd1b0f18561"). InnerVolumeSpecName "kube-api-access-6w55k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:32.662518 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.662491    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-plugins-config-volume\") pod \"b08d7a98-4183-4f36-918d-3a5df4aaa0ab\" (UID: \"b08d7a98-4183-4f36-918d-3a5df4aaa0ab\") "
Oct 30 03:31:32.662667 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.662526    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-kube-api-access-w7r86\" (UniqueName: \"kubernetes.io/projected/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-kube-api-access-w7r86\") pod \"b08d7a98-4183-4f36-918d-3a5df4aaa0ab\" (UID: \"b08d7a98-4183-4f36-918d-3a5df4aaa0ab\") "
Oct 30 03:31:32.662765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.662752    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a5a139-8b79-485f-9392-4fd1b0f18561-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:32.662833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.662766    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w55k\" (UniqueName: \"kubernetes.io/projected/e3a5a139-8b79-485f-9392-4fd1b0f18561-kube-api-access-6w55k\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:32.662833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.662819    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-plugins-config-volume" pod "b08d7a98-4183-4f36-918d-3a5df4aaa0ab" (UID: "b08d7a98-4183-4f36-918d-3a5df4aaa0ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:32.664651 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.664627    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-kube-api-access-w7r86" pod "b08d7a98-4183-4f36-918d-3a5df4aaa0ab" (UID: "b08d7a98-4183-4f36-918d-3a5df4aaa0ab"). InnerVolumeSpecName "kube-api-access-w7r86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:32.763929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.763902    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:32.763929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.763930    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7r86\" (UniqueName: \"kubernetes.io/projected/b08d7a98-4183-4f36-918d-3a5df4aaa0ab-kube-api-access-w7r86\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:32.765022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.764999    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:32.782422 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.782399    5061 generic.go:356] "Generic (PLEG): container finished" podID="c2fedcd9-ed61-4532-ae59-8583c533412e" containerID="3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76" exitCode=0
Oct 30 03:31:32.782526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.782447    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"
Oct 30 03:31:32.782526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.782478    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h" event={"ID":"c2fedcd9-ed61-4532-ae59-8583c533412e","Type":"ContainerDied","Data":"3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76"}
Oct 30 03:31:32.782628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.782535    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h" event={"ID":"c2fedcd9-ed61-4532-ae59-8583c533412e","Type":"ContainerDied","Data":"fb13a4a2d3b9a1b511ffba1a79b5cc42bd06563bd408ec2a9c93a55823b750eb"}
Oct 30 03:31:32.782628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.782568    5061 scope.go:122] "RemoveContainer" containerID="3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76"
Oct 30 03:31:32.784169 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.784143    5061 generic.go:356] "Generic (PLEG): container finished" podID="b08d7a98-4183-4f36-918d-3a5df4aaa0ab" containerID="1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4" exitCode=0
Oct 30 03:31:32.784292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.784190    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"
Oct 30 03:31:32.784292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.784239    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565" event={"ID":"b08d7a98-4183-4f36-918d-3a5df4aaa0ab","Type":"ContainerDied","Data":"1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4"}
Oct 30 03:31:32.784292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.784277    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565" event={"ID":"b08d7a98-4183-4f36-918d-3a5df4aaa0ab","Type":"ContainerDied","Data":"707e597ef7b884186dd7086ab6ec8e11a55dd87d4ccd57f57f6357e359d843c2"}
Oct 30 03:31:32.788086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.788062    5061 generic.go:356] "Generic (PLEG): container finished" podID="e3a5a139-8b79-485f-9392-4fd1b0f18561" containerID="6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00" exitCode=0
Oct 30 03:31:32.788194 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.788103    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"
Oct 30 03:31:32.788266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.788102    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk" event={"ID":"e3a5a139-8b79-485f-9392-4fd1b0f18561","Type":"ContainerDied","Data":"6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00"}
Oct 30 03:31:32.788266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.788238    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk" event={"ID":"e3a5a139-8b79-485f-9392-4fd1b0f18561","Type":"ContainerDied","Data":"ea8a2501352c36eaf7bb6064ff7912107518eba421c7c545059dc81fc19d26a7"}
Oct 30 03:31:32.792663 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.792645    5061 scope.go:122] "RemoveContainer" containerID="3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76"
Oct 30 03:31:32.792970 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:31:32.792949    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76\": container with ID starting with 3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76 not found: ID does not exist" containerID="3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76"
Oct 30 03:31:32.793072 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.792976    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76"} err="failed to get container status \"3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76\": rpc error: code = NotFound desc = could not find container \"3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76\": container with ID starting with 3670a09bb1d586198f86a45adcb51a41d9f6869578382f842b775c484ce67e76 not found: ID does not exist"
Oct 30 03:31:32.793072 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.792991    5061 scope.go:122] "RemoveContainer" containerID="1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4"
Oct 30 03:31:32.804685 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.804666    5061 scope.go:122] "RemoveContainer" containerID="1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4"
Oct 30 03:31:32.805411 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:31:32.805392    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4\": container with ID starting with 1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4 not found: ID does not exist" containerID="1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4"
Oct 30 03:31:32.805488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.805417    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4"} err="failed to get container status \"1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4\": rpc error: code = NotFound desc = could not find container \"1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4\": container with ID starting with 1cb31c1680ad37524ccc1d19c8e202baf4b3d4f9bd62ae5860bcd8c0ac3d8be4 not found: ID does not exist"
Oct 30 03:31:32.805488 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.805433    5061 scope.go:122] "RemoveContainer" containerID="6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00"
Oct 30 03:31:32.813445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.813423    5061 scope.go:122] "RemoveContainer" containerID="6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00"
Oct 30 03:31:32.813704 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:31:32.813686    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00\": container with ID starting with 6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00 not found: ID does not exist" containerID="6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00"
Oct 30 03:31:32.813771 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.813714    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00"} err="failed to get container status \"6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00\": rpc error: code = NotFound desc = could not find container \"6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00\": container with ID starting with 6dc8b6aa7783723b08ef86e135ade6b7bd72778407a4d88a662e3935b6d05d00 not found: ID does not exist"
Oct 30 03:31:32.854373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.854341    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"]
Oct 30 03:31:32.858092 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.858060    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f72d5-fc95f6f99-hn565"]
Oct 30 03:31:32.881797 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.881774    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:32.885641 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.885621    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"]
Oct 30 03:31:32.887947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.887923    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f72d5-5c784c8d8b-v7svk"]
Oct 30 03:31:32.917383 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.917353    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"]
Oct 30 03:31:32.921155 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:32.921129    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f72d5-7dcbc89c65-vfs5h"]
Oct 30 03:31:33.367422 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:33.367387    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:34.529809 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:34.529745    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:34.845936 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:34.845906    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08d7a98-4183-4f36-918d-3a5df4aaa0ab" path="/var/lib/kubelet/pods/b08d7a98-4183-4f36-918d-3a5df4aaa0ab/volumes"
Oct 30 03:31:34.846323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:34.846310    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2fedcd9-ed61-4532-ae59-8583c533412e" path="/var/lib/kubelet/pods/c2fedcd9-ed61-4532-ae59-8583c533412e/volumes"
Oct 30 03:31:34.846655 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:34.846643    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a5a139-8b79-485f-9392-4fd1b0f18561" path="/var/lib/kubelet/pods/e3a5a139-8b79-485f-9392-4fd1b0f18561/volumes"
Oct 30 03:31:36.097988 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:36.097950    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:37.616992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:37.616953    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:39.070709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:39.070675    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:39.626673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:39.626637    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:39.842811 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:39.842782    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:39.852716 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:39.852687    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"]
Oct 30 03:31:39.852990 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:39.852961    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy" containerID="cri-o://4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536" gracePeriod=30
Oct 30 03:31:39.964964 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:39.964874    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:40.791707 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:40.791679    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:42.542241 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:42.542203    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:44.096105 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:44.096068    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:45.116122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.116095    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:31:45.189120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189054    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-certs\") pod \"491108be-282e-45ef-ac58-a9404fc21748\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") "
Oct 30 03:31:45.189120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189097    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-data\") pod \"491108be-282e-45ef-ac58-a9404fc21748\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") "
Oct 30 03:31:45.189340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189133    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/491108be-282e-45ef-ac58-a9404fc21748-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491108be-282e-45ef-ac58-a9404fc21748-istio-podinfo\") pod \"491108be-282e-45ef-ac58-a9404fc21748\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") "
Oct 30 03:31:45.189340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189189    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-istio-token\" (UniqueName: \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-istio-token\") pod \"491108be-282e-45ef-ac58-a9404fc21748\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") "
Oct 30 03:31:45.189340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189227    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-kube-api-access-p2smg\" (UniqueName: \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-kube-api-access-p2smg\") pod \"491108be-282e-45ef-ac58-a9404fc21748\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") "
Oct 30 03:31:45.189340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189269    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/491108be-282e-45ef-ac58-a9404fc21748-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491108be-282e-45ef-ac58-a9404fc21748-istiod-ca-cert\") pod \"491108be-282e-45ef-ac58-a9404fc21748\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") "
Oct 30 03:31:45.189340 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189327    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-socket\") pod \"491108be-282e-45ef-ac58-a9404fc21748\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") "
Oct 30 03:31:45.189619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189353    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-envoy\") pod \"491108be-282e-45ef-ac58-a9404fc21748\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") "
Oct 30 03:31:45.189619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189368    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-certs" pod "491108be-282e-45ef-ac58-a9404fc21748" (UID: "491108be-282e-45ef-ac58-a9404fc21748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:45.189619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189392    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-credential-socket\") pod \"491108be-282e-45ef-ac58-a9404fc21748\" (UID: \"491108be-282e-45ef-ac58-a9404fc21748\") "
Oct 30 03:31:45.189619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189432    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-data" pod "491108be-282e-45ef-ac58-a9404fc21748" (UID: "491108be-282e-45ef-ac58-a9404fc21748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:45.189619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189606    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-credential-socket" pod "491108be-282e-45ef-ac58-a9404fc21748" (UID: "491108be-282e-45ef-ac58-a9404fc21748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:45.189941 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189912    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:45.189941 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189934    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:45.190057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189952    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:45.190057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.189989    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-socket" pod "491108be-282e-45ef-ac58-a9404fc21748" (UID: "491108be-282e-45ef-ac58-a9404fc21748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:45.190057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.190036    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491108be-282e-45ef-ac58-a9404fc21748-istiod-ca-cert" pod "491108be-282e-45ef-ac58-a9404fc21748" (UID: "491108be-282e-45ef-ac58-a9404fc21748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:45.191722 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.191690    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/491108be-282e-45ef-ac58-a9404fc21748-istio-podinfo" pod "491108be-282e-45ef-ac58-a9404fc21748" (UID: "491108be-282e-45ef-ac58-a9404fc21748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:31:45.191885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.191752    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-kube-api-access-p2smg" pod "491108be-282e-45ef-ac58-a9404fc21748" (UID: "491108be-282e-45ef-ac58-a9404fc21748"). InnerVolumeSpecName "kube-api-access-p2smg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:45.191885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.191838    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-envoy" pod "491108be-282e-45ef-ac58-a9404fc21748" (UID: "491108be-282e-45ef-ac58-a9404fc21748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:45.192398 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.192379    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-istio-token" pod "491108be-282e-45ef-ac58-a9404fc21748" (UID: "491108be-282e-45ef-ac58-a9404fc21748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:45.291337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.291311    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:45.291337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.291337    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2smg\" (UniqueName: \"kubernetes.io/projected/491108be-282e-45ef-ac58-a9404fc21748-kube-api-access-p2smg\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:45.291503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.291348    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491108be-282e-45ef-ac58-a9404fc21748-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:45.291503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.291358    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:45.291503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.291366    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491108be-282e-45ef-ac58-a9404fc21748-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:45.291503 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.291375    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491108be-282e-45ef-ac58-a9404fc21748-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:31:45.736429 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.736396    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:45.853995 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.853963    5061 generic.go:356] "Generic (PLEG): container finished" podID="491108be-282e-45ef-ac58-a9404fc21748" containerID="4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536" exitCode=0
Oct 30 03:31:45.854190 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.854016    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"
Oct 30 03:31:45.854190 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.854048    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" event={"ID":"491108be-282e-45ef-ac58-a9404fc21748","Type":"ContainerDied","Data":"4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536"}
Oct 30 03:31:45.854190 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.854104    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g" event={"ID":"491108be-282e-45ef-ac58-a9404fc21748","Type":"ContainerDied","Data":"bbc7ab43723328dcf1c90de42ad22c3a7436f3a138c7081c8a5173a21d8ec1af"}
Oct 30 03:31:45.854190 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.854131    5061 scope.go:122] "RemoveContainer" containerID="4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536"
Oct 30 03:31:45.868220 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.868158    5061 scope.go:122] "RemoveContainer" containerID="84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227"
Oct 30 03:31:45.869754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.869724    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:45.877086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.877065    5061 scope.go:122] "RemoveContainer" containerID="4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536"
Oct 30 03:31:45.877363 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:31:45.877342    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536\": container with ID starting with 4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536 not found: ID does not exist" containerID="4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536"
Oct 30 03:31:45.877411 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.877372    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536"} err="failed to get container status \"4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536\": rpc error: code = NotFound desc = could not find container \"4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536\": container with ID starting with 4305615693b9f66062567d3828760ef97df13d06bc82e080d34d2efc0f9a7536 not found: ID does not exist"
Oct 30 03:31:45.877411 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.877389    5061 scope.go:122] "RemoveContainer" containerID="84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227"
Oct 30 03:31:45.877707 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:31:45.877661    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227\": container with ID starting with 84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227 not found: ID does not exist" containerID="84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227"
Oct 30 03:31:45.877767 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.877730    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227"} err="failed to get container status \"84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227\": rpc error: code = NotFound desc = could not find container \"84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227\": container with ID starting with 84eedeb9dc38c7320bc98dcfd7341034ab486e02c89495883e17153d709dc227 not found: ID does not exist"
Oct 30 03:31:45.911280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.911255    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"]
Oct 30 03:31:45.921550 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:45.921523    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3f72d5-openshift-default-7df4497f55-d5r7g"]
Oct 30 03:31:46.845607 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:46.845564    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="491108be-282e-45ef-ac58-a9404fc21748" path="/var/lib/kubelet/pods/491108be-282e-45ef-ac58-a9404fc21748/volumes"
Oct 30 03:31:47.235513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:47.235433    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:48.920242 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:48.920201    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:50.431054 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:50.431017    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:50.909982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:50.909936    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:51.048409 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:51.048365    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:52.207720 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:52.207688    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:53.432222 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:53.432142    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:53.880829 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:53.880742    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:53.998453 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:53.998361    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:54.533534 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:54.533444    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:55.466252 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:31:55.466157    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:31:55.602505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:55.602421    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:57.295207 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:57.295170    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:58.779888 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:31:58.779835    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:00.530940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:00.530895    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:00.925215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:00.925179    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:01.146646 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:01.146610    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:01.205799 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:01.205711    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"]
Oct 30 03:32:01.206038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:01.206005    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" podUID="2a7fd568-5977-42fc-987d-61af621d4d38" containerName="istio-proxy" containerID="cri-o://626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874" gracePeriod=30
Oct 30 03:32:01.284645 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:01.284619    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:04.049113 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:04.049071    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:04.220914 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:04.220874    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:05.233245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:05.233203    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:06.461005 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.460980    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:32:06.625947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.625917    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-certs\") pod \"2a7fd568-5977-42fc-987d-61af621d4d38\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") "
Oct 30 03:32:06.625947 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.625949    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-socket\") pod \"2a7fd568-5977-42fc-987d-61af621d4d38\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") "
Oct 30 03:32:06.626175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626022    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-credential-socket\") pod \"2a7fd568-5977-42fc-987d-61af621d4d38\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") "
Oct 30 03:32:06.626175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626041    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a7fd568-5977-42fc-987d-61af621d4d38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7fd568-5977-42fc-987d-61af621d4d38-istio-podinfo\") pod \"2a7fd568-5977-42fc-987d-61af621d4d38\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") "
Oct 30 03:32:06.626175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626064    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7fd568-5977-42fc-987d-61af621d4d38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7fd568-5977-42fc-987d-61af621d4d38-istiod-ca-cert\") pod \"2a7fd568-5977-42fc-987d-61af621d4d38\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") "
Oct 30 03:32:06.626175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626086    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-envoy\") pod \"2a7fd568-5977-42fc-987d-61af621d4d38\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") "
Oct 30 03:32:06.626175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626121    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-istio-token\" (UniqueName: \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-istio-token\") pod \"2a7fd568-5977-42fc-987d-61af621d4d38\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") "
Oct 30 03:32:06.626175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626158    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-data\") pod \"2a7fd568-5977-42fc-987d-61af621d4d38\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") "
Oct 30 03:32:06.626507 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626206    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-kube-api-access-ss79f\" (UniqueName: \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-kube-api-access-ss79f\") pod \"2a7fd568-5977-42fc-987d-61af621d4d38\" (UID: \"2a7fd568-5977-42fc-987d-61af621d4d38\") "
Oct 30 03:32:06.626507 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626222    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-certs" pod "2a7fd568-5977-42fc-987d-61af621d4d38" (UID: "2a7fd568-5977-42fc-987d-61af621d4d38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:06.626507 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626299    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-socket" pod "2a7fd568-5977-42fc-987d-61af621d4d38" (UID: "2a7fd568-5977-42fc-987d-61af621d4d38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:06.626507 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626470    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-data" pod "2a7fd568-5977-42fc-987d-61af621d4d38" (UID: "2a7fd568-5977-42fc-987d-61af621d4d38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:06.626741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626562    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7fd568-5977-42fc-987d-61af621d4d38-istiod-ca-cert" pod "2a7fd568-5977-42fc-987d-61af621d4d38" (UID: "2a7fd568-5977-42fc-987d-61af621d4d38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:06.626741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626582    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.626741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626607    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.626741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626620    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.626741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.626684    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-credential-socket" pod "2a7fd568-5977-42fc-987d-61af621d4d38" (UID: "2a7fd568-5977-42fc-987d-61af621d4d38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:06.628282 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.628261    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-envoy" pod "2a7fd568-5977-42fc-987d-61af621d4d38" (UID: "2a7fd568-5977-42fc-987d-61af621d4d38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:06.628504 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.628481    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a7fd568-5977-42fc-987d-61af621d4d38-istio-podinfo" pod "2a7fd568-5977-42fc-987d-61af621d4d38" (UID: "2a7fd568-5977-42fc-987d-61af621d4d38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:32:06.628788 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.628761    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-istio-token" pod "2a7fd568-5977-42fc-987d-61af621d4d38" (UID: "2a7fd568-5977-42fc-987d-61af621d4d38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:06.628875 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.628807    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-kube-api-access-ss79f" pod "2a7fd568-5977-42fc-987d-61af621d4d38" (UID: "2a7fd568-5977-42fc-987d-61af621d4d38"). InnerVolumeSpecName "kube-api-access-ss79f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:06.728149 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.728106    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.728149 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.728146    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7fd568-5977-42fc-987d-61af621d4d38-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.728149 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.728156    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7fd568-5977-42fc-987d-61af621d4d38-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.728433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.728165    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7fd568-5977-42fc-987d-61af621d4d38-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.728433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.728177    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.728433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.728189    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss79f\" (UniqueName: \"kubernetes.io/projected/2a7fd568-5977-42fc-987d-61af621d4d38-kube-api-access-ss79f\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:06.879062 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.878991    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:06.965375 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.965340    5061 generic.go:356] "Generic (PLEG): container finished" podID="2a7fd568-5977-42fc-987d-61af621d4d38" containerID="626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874" exitCode=0
Oct 30 03:32:06.965563 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.965382    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"
Oct 30 03:32:06.965563 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.965423    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" event={"ID":"2a7fd568-5977-42fc-987d-61af621d4d38","Type":"ContainerDied","Data":"626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874"}
Oct 30 03:32:06.965563 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.965484    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42" event={"ID":"2a7fd568-5977-42fc-987d-61af621d4d38","Type":"ContainerDied","Data":"9ed4e368aefc1b3e7d0c2366ae870bef5ab65e4770c3a6a2d1b33f02762c2afb"}
Oct 30 03:32:06.965563 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.965508    5061 scope.go:122] "RemoveContainer" containerID="626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874"
Oct 30 03:32:06.974898 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.974882    5061 scope.go:122] "RemoveContainer" containerID="626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874"
Oct 30 03:32:06.975181 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:32:06.975162    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874\": container with ID starting with 626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874 not found: ID does not exist" containerID="626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874"
Oct 30 03:32:06.975248 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:06.975191    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874"} err="failed to get container status \"626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874\": rpc error: code = NotFound desc = could not find container \"626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874\": container with ID starting with 626d977736d15aeb6d1728834bfe2077dbe5439f359876ba97d3e4a20a5d7874 not found: ID does not exist"
Oct 30 03:32:07.011921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:07.011896    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"]
Oct 30 03:32:07.016208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:07.016188    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-d823d1-openshift-default-566cfb7479-dvw42"]
Oct 30 03:32:08.323029 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:08.322995    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:08.846485 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:08.846454    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7fd568-5977-42fc-987d-61af621d4d38" path="/var/lib/kubelet/pods/2a7fd568-5977-42fc-987d-61af621d4d38/volumes"
Oct 30 03:32:09.879892 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:09.879838    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:11.339758 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:11.339720    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:12.329336 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:12.329296    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:12.450157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:12.450120    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:12.594885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:12.594776    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:13.152622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:13.152588    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:13.573001 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:13.572969    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:13.690336 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:13.690298    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:14.204221 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:14.204181    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:14.964307 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:14.964275    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:16.530316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:16.530283    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:18.166267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:18.166221    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:19.104904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:19.104868    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:19.779318 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:19.779284    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:20.495514 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:20.495485    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:20.640586 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:20.640552    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"]
Oct 30 03:32:20.640879 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:20.640836    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf" podUID="e28f99f1-cc49-4c8b-8a34-7440b8068712" containerName="istio-proxy" containerID="cri-o://e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af" gracePeriod=30
Oct 30 03:32:20.645826 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:20.645801    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:20.771841 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:20.771759    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:21.184546 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:21.184511    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:21.391946 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:21.391873    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:23.008525 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:23.008494    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:24.494219 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:24.494184    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:25.446161 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:32:25.446122    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:32:25.906501 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:25.906472    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:32:25.977741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:25.977683    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:26.008083 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.008049    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-data\") pod \"e28f99f1-cc49-4c8b-8a34-7440b8068712\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") "
Oct 30 03:32:26.008265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.008097    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-podinfo\") pod \"e28f99f1-cc49-4c8b-8a34-7440b8068712\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") "
Oct 30 03:32:26.008265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.008131    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-socket\") pod \"e28f99f1-cc49-4c8b-8a34-7440b8068712\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") "
Oct 30 03:32:26.008265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.008206    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28f99f1-cc49-4c8b-8a34-7440b8068712-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28f99f1-cc49-4c8b-8a34-7440b8068712-istiod-ca-cert\") pod \"e28f99f1-cc49-4c8b-8a34-7440b8068712\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") "
Oct 30 03:32:26.008265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.008245    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-token\" (UniqueName: \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-token\") pod \"e28f99f1-cc49-4c8b-8a34-7440b8068712\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") "
Oct 30 03:32:26.008510 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.008273    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-certs\") pod \"e28f99f1-cc49-4c8b-8a34-7440b8068712\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") "
Oct 30 03:32:26.008840 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.008593    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-certs" pod "e28f99f1-cc49-4c8b-8a34-7440b8068712" (UID: "e28f99f1-cc49-4c8b-8a34-7440b8068712"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.008874    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-data" pod "e28f99f1-cc49-4c8b-8a34-7440b8068712" (UID: "e28f99f1-cc49-4c8b-8a34-7440b8068712"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.009049    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-envoy\") pod \"e28f99f1-cc49-4c8b-8a34-7440b8068712\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") "
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.008899    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28f99f1-cc49-4c8b-8a34-7440b8068712-istiod-ca-cert" pod "e28f99f1-cc49-4c8b-8a34-7440b8068712" (UID: "e28f99f1-cc49-4c8b-8a34-7440b8068712"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.009024    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-socket" pod "e28f99f1-cc49-4c8b-8a34-7440b8068712" (UID: "e28f99f1-cc49-4c8b-8a34-7440b8068712"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.009202    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-credential-socket\") pod \"e28f99f1-cc49-4c8b-8a34-7440b8068712\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") "
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.009258    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-kube-api-access-ckv48\" (UniqueName: \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-kube-api-access-ckv48\") pod \"e28f99f1-cc49-4c8b-8a34-7440b8068712\" (UID: \"e28f99f1-cc49-4c8b-8a34-7440b8068712\") "
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.009467    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-credential-socket" pod "e28f99f1-cc49-4c8b-8a34-7440b8068712" (UID: "e28f99f1-cc49-4c8b-8a34-7440b8068712"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.009598    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.009617    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.009631    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.009644    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28f99f1-cc49-4c8b-8a34-7440b8068712-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.009681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.009658    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.010982 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.010960    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-token" pod "e28f99f1-cc49-4c8b-8a34-7440b8068712" (UID: "e28f99f1-cc49-4c8b-8a34-7440b8068712"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:26.011382 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.011363    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-podinfo" pod "e28f99f1-cc49-4c8b-8a34-7440b8068712" (UID: "e28f99f1-cc49-4c8b-8a34-7440b8068712"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:32:26.011466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.011365    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-envoy" pod "e28f99f1-cc49-4c8b-8a34-7440b8068712" (UID: "e28f99f1-cc49-4c8b-8a34-7440b8068712"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:26.011728 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.011708    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-kube-api-access-ckv48" pod "e28f99f1-cc49-4c8b-8a34-7440b8068712" (UID: "e28f99f1-cc49-4c8b-8a34-7440b8068712"). InnerVolumeSpecName "kube-api-access-ckv48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:26.056623 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.056598    5061 generic.go:356] "Generic (PLEG): container finished" podID="e28f99f1-cc49-4c8b-8a34-7440b8068712" containerID="e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af" exitCode=0
Oct 30 03:32:26.056715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.056647    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"
Oct 30 03:32:26.056715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.056687    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf" event={"ID":"e28f99f1-cc49-4c8b-8a34-7440b8068712","Type":"ContainerDied","Data":"e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af"}
Oct 30 03:32:26.056798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.056740    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf" event={"ID":"e28f99f1-cc49-4c8b-8a34-7440b8068712","Type":"ContainerDied","Data":"6d04b5198e5f0279407a56766f2c16eb1ae4a29a49ac6624bd21895f4a22d3d5"}
Oct 30 03:32:26.056798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.056762    5061 scope.go:122] "RemoveContainer" containerID="e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af"
Oct 30 03:32:26.065925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.065911    5061 scope.go:122] "RemoveContainer" containerID="e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af"
Oct 30 03:32:26.066204 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:32:26.066187    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af\": container with ID starting with e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af not found: ID does not exist" containerID="e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af"
Oct 30 03:32:26.066258 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.066211    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af"} err="failed to get container status \"e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af\": rpc error: code = NotFound desc = could not find container \"e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af\": container with ID starting with e0a050f6556eff3d1e22bc69dd2daa42ad70d5e18bd3f8567427626c1a6e41af not found: ID does not exist"
Oct 30 03:32:26.104023 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.104000    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"]
Oct 30 03:32:26.106381 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.106360    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-874ab5-openshift-default-554c6b8cc5-kh9rf"]
Oct 30 03:32:26.110100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.110080    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckv48\" (UniqueName: \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-kube-api-access-ckv48\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.110182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.110101    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.110182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.110112    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.110182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.110122    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28f99f1-cc49-4c8b-8a34-7440b8068712-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:26.123378 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.123355    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"]
Oct 30 03:32:26.135219 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.135198    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"]
Oct 30 03:32:26.234115 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.234085    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:26.845554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:26.845524    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e28f99f1-cc49-4c8b-8a34-7440b8068712" path="/var/lib/kubelet/pods/e28f99f1-cc49-4c8b-8a34-7440b8068712/volumes"
Oct 30 03:32:27.663424 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:27.663394    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:29.188068 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:29.188032    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:30.717722 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:30.717691    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:31.898540 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:31.898509    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:32.015300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:32.015266    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:32.283494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:32.283419    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:41.144712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.144626    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main" containerID="cri-o://67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa" gracePeriod=45
Oct 30 03:32:41.154142 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.154095    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main" containerID="cri-o://fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b" gracePeriod=45
Oct 30 03:32:41.405614 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.405555    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 03:32:41.425786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.425760    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 03:32:41.548224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548179    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-tmp-dir\") pod \"4f6e7199-66a6-4043-9bdd-ca245968dfef\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") "
Oct 30 03:32:41.548402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548248    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f6e7199-66a6-4043-9bdd-ca245968dfef-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6e7199-66a6-4043-9bdd-ca245968dfef-tls-certs\") pod \"4f6e7199-66a6-4043-9bdd-ca245968dfef\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") "
Oct 30 03:32:41.548402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548331    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6e7199-66a6-4043-9bdd-ca245968dfef-kube-api-access-gksxr\" (UniqueName: \"kubernetes.io/projected/4f6e7199-66a6-4043-9bdd-ca245968dfef-kube-api-access-gksxr\") pod \"4f6e7199-66a6-4043-9bdd-ca245968dfef\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") "
Oct 30 03:32:41.548402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548367    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-dshm\") pod \"4f6e7199-66a6-4043-9bdd-ca245968dfef\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") "
Oct 30 03:32:41.548596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548424    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tls-certs\") pod \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") "
Oct 30 03:32:41.548596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548435    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-tmp-dir" pod "4f6e7199-66a6-4043-9bdd-ca245968dfef" (UID: "4f6e7199-66a6-4043-9bdd-ca245968dfef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.548596 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548522    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-model-cache\") pod \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") "
Oct 30 03:32:41.548759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548626    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-dshm\") pod \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") "
Oct 30 03:32:41.548759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548699    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-kube-api-access-bvrnk\" (UniqueName: \"kubernetes.io/projected/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-kube-api-access-bvrnk\") pod \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") "
Oct 30 03:32:41.548759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548741    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-home\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-home\") pod \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") "
Oct 30 03:32:41.548966 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548767    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-model-cache\") pod \"4f6e7199-66a6-4043-9bdd-ca245968dfef\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") "
Oct 30 03:32:41.548966 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548811    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tmp-dir\") pod \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\" (UID: \"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb\") "
Oct 30 03:32:41.548966 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548826    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-model-cache" pod "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" (UID: "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.548966 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.548861    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-home\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-home\") pod \"4f6e7199-66a6-4043-9bdd-ca245968dfef\" (UID: \"4f6e7199-66a6-4043-9bdd-ca245968dfef\") "
Oct 30 03:32:41.549249 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.549069    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-home" pod "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" (UID: "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.549249 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.549217    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.549249 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.549238    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.549419 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.549254    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.549502 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.549480    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-model-cache" pod "4f6e7199-66a6-4043-9bdd-ca245968dfef" (UID: "4f6e7199-66a6-4043-9bdd-ca245968dfef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.549697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.549676    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tmp-dir" pod "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" (UID: "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.549911 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.549891    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-home" pod "4f6e7199-66a6-4043-9bdd-ca245968dfef" (UID: "4f6e7199-66a6-4043-9bdd-ca245968dfef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.550983 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.550952    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tls-certs" pod "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" (UID: "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:41.551086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.551019    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6e7199-66a6-4043-9bdd-ca245968dfef-tls-certs" pod "4f6e7199-66a6-4043-9bdd-ca245968dfef" (UID: "4f6e7199-66a6-4043-9bdd-ca245968dfef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:41.551086 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.551017    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-dshm" pod "4f6e7199-66a6-4043-9bdd-ca245968dfef" (UID: "4f6e7199-66a6-4043-9bdd-ca245968dfef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.551168 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.551110    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6e7199-66a6-4043-9bdd-ca245968dfef-kube-api-access-gksxr" pod "4f6e7199-66a6-4043-9bdd-ca245968dfef" (UID: "4f6e7199-66a6-4043-9bdd-ca245968dfef"). InnerVolumeSpecName "kube-api-access-gksxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:41.551560 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.551546    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-dshm" pod "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" (UID: "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:41.551780 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.551765    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-kube-api-access-bvrnk" pod "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" (UID: "6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb"). InnerVolumeSpecName "kube-api-access-bvrnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:41.650354 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.650317    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gksxr\" (UniqueName: \"kubernetes.io/projected/4f6e7199-66a6-4043-9bdd-ca245968dfef-kube-api-access-gksxr\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.650354 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.650348    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.650354 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.650359    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.650668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.650368    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.650668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.650376    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvrnk\" (UniqueName: \"kubernetes.io/projected/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-kube-api-access-bvrnk\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.650668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.650384    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.650668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.650392    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.650668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.650399    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6e7199-66a6-4043-9bdd-ca245968dfef-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:41.650668 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:41.650406    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6e7199-66a6-4043-9bdd-ca245968dfef-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:42.137412 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.137373    5061 generic.go:356] "Generic (PLEG): container finished" podID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerID="67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa" exitCode=0
Oct 30 03:32:42.137606 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.137431    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"
Oct 30 03:32:42.137606 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.137452    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" event={"ID":"4f6e7199-66a6-4043-9bdd-ca245968dfef","Type":"ContainerDied","Data":"67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa"}
Oct 30 03:32:42.137606 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.137499    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9" event={"ID":"4f6e7199-66a6-4043-9bdd-ca245968dfef","Type":"ContainerDied","Data":"b61cf741dbaeb4667d730de0a130af07b9eeed9a9f53ce872188970a27d16b75"}
Oct 30 03:32:42.137606 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.137524    5061 scope.go:122] "RemoveContainer" containerID="67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa"
Oct 30 03:32:42.138891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.138866    5061 generic.go:356] "Generic (PLEG): container finished" podID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerID="fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b" exitCode=0
Oct 30 03:32:42.138996 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.138915    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"
Oct 30 03:32:42.139041 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.138903    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" event={"ID":"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb","Type":"ContainerDied","Data":"fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b"}
Oct 30 03:32:42.139097 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.139033    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc" event={"ID":"6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb","Type":"ContainerDied","Data":"c9b857c779ae2b40331a325964a58541f51e59c4babc0fe6d091f415a68732ed"}
Oct 30 03:32:42.146820 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.146490    5061 scope.go:122] "RemoveContainer" containerID="67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa"
Oct 30 03:32:42.146820 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:32:42.146783    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa\": container with ID starting with 67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa not found: ID does not exist" containerID="67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa"
Oct 30 03:32:42.147314 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.146823    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa"} err="failed to get container status \"67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa\": rpc error: code = NotFound desc = could not find container \"67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa\": container with ID starting with 67034402f0a169e99568061e904ac1f79d4d1b20ccd1ff4130133f74edbf64fa not found: ID does not exist"
Oct 30 03:32:42.147314 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.146869    5061 scope.go:122] "RemoveContainer" containerID="fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b"
Oct 30 03:32:42.155064 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.155045    5061 scope.go:122] "RemoveContainer" containerID="fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b"
Oct 30 03:32:42.155326 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:32:42.155307    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b\": container with ID starting with fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b not found: ID does not exist" containerID="fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b"
Oct 30 03:32:42.155382 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.155333    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b"} err="failed to get container status \"fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b\": rpc error: code = NotFound desc = could not find container \"fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b\": container with ID starting with fb06d327abdfdc6b62edd236fd04e40421d0d560d9725708982b47774734960b not found: ID does not exist"
Oct 30 03:32:42.182709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.182687    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"]
Oct 30 03:32:42.192767 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.192747    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/premium-w0-aeff1d-kserve-7f76d8fc6c-7xsqc"]
Oct 30 03:32:42.227498 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.227454    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"]
Oct 30 03:32:42.227665 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.227520    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-aeff1d/distinct-w0-aeff1d-kserve-6655df77c4-kctg9"]
Oct 30 03:32:42.846858 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.846815    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" path="/var/lib/kubelet/pods/4f6e7199-66a6-4043-9bdd-ca245968dfef/volumes"
Oct 30 03:32:42.847246 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:42.847232    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" path="/var/lib/kubelet/pods/6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb/volumes"
Oct 30 03:32:48.601518 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:48.601483    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"]
Oct 30 03:32:48.602182 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:48.601730    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq" podUID="d1a22ae9-6252-4363-b95b-fa3708998135" containerName="maas-api" containerID="cri-o://8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e" gracePeriod=30
Oct 30 03:32:48.698061 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:48.697996    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:48.858787 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:48.858717    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 03:32:49.018108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.018075    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a22ae9-6252-4363-b95b-fa3708998135-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a22ae9-6252-4363-b95b-fa3708998135-openshift-service-ca\") pod \"d1a22ae9-6252-4363-b95b-fa3708998135\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") "
Oct 30 03:32:49.018305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.018119    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a22ae9-6252-4363-b95b-fa3708998135-kube-api-access-gsq88\" (UniqueName: \"kubernetes.io/projected/d1a22ae9-6252-4363-b95b-fa3708998135-kube-api-access-gsq88\") pod \"d1a22ae9-6252-4363-b95b-fa3708998135\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") "
Oct 30 03:32:49.018305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.018157    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-maas-api-tls\") pod \"d1a22ae9-6252-4363-b95b-fa3708998135\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") "
Oct 30 03:32:49.018305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.018183    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-metrics-certs\") pod \"d1a22ae9-6252-4363-b95b-fa3708998135\" (UID: \"d1a22ae9-6252-4363-b95b-fa3708998135\") "
Oct 30 03:32:49.018545 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.018515    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a22ae9-6252-4363-b95b-fa3708998135-openshift-service-ca" pod "d1a22ae9-6252-4363-b95b-fa3708998135" (UID: "d1a22ae9-6252-4363-b95b-fa3708998135"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:49.018821 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.018798    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a22ae9-6252-4363-b95b-fa3708998135-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:49.020990 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.020961    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-metrics-certs" pod "d1a22ae9-6252-4363-b95b-fa3708998135" (UID: "d1a22ae9-6252-4363-b95b-fa3708998135"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:49.021080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.020991    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-maas-api-tls" pod "d1a22ae9-6252-4363-b95b-fa3708998135" (UID: "d1a22ae9-6252-4363-b95b-fa3708998135"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:49.021127 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.021102    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a22ae9-6252-4363-b95b-fa3708998135-kube-api-access-gsq88" pod "d1a22ae9-6252-4363-b95b-fa3708998135" (UID: "d1a22ae9-6252-4363-b95b-fa3708998135"). InnerVolumeSpecName "kube-api-access-gsq88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:49.106838 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.106800    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:49.119558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.119498    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsq88\" (UniqueName: \"kubernetes.io/projected/d1a22ae9-6252-4363-b95b-fa3708998135-kube-api-access-gsq88\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:49.119558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.119522    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:49.119558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.119531    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a22ae9-6252-4363-b95b-fa3708998135-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:32:49.174272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.174244    5061 generic.go:356] "Generic (PLEG): container finished" podID="d1a22ae9-6252-4363-b95b-fa3708998135" containerID="8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e" exitCode=0
Oct 30 03:32:49.174404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.174289    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"
Oct 30 03:32:49.174404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.174323    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq" event={"ID":"d1a22ae9-6252-4363-b95b-fa3708998135","Type":"ContainerDied","Data":"8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e"}
Oct 30 03:32:49.174404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.174380    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq" event={"ID":"d1a22ae9-6252-4363-b95b-fa3708998135","Type":"ContainerDied","Data":"7914e116278fb405e8a1f4c9bf0a38c4aee8d77529cf2687f6fb3e62c413e9bf"}
Oct 30 03:32:49.174404 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.174395    5061 scope.go:122] "RemoveContainer" containerID="8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e"
Oct 30 03:32:49.183794 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.183778    5061 scope.go:122] "RemoveContainer" containerID="8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e"
Oct 30 03:32:49.184085 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:32:49.184066    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e\": container with ID starting with 8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e not found: ID does not exist" containerID="8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e"
Oct 30 03:32:49.184150 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.184093    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e"} err="failed to get container status \"8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e\": rpc error: code = NotFound desc = could not find container \"8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e\": container with ID starting with 8543e9e9faaef5705026dec08b98771837ebca49fe775f7b39e49ebd3f1dbf3e not found: ID does not exist"
Oct 30 03:32:49.222446 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.222423    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"]
Oct 30 03:32:49.228706 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.228680    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-aeff1d-5d4d99fd75-hcbgq"]
Oct 30 03:32:49.230568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.230542    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:49.765920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:49.765864    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:50.846224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:50.846192    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a22ae9-6252-4363-b95b-fa3708998135" path="/var/lib/kubelet/pods/d1a22ae9-6252-4363-b95b-fa3708998135/volumes"
Oct 30 03:32:51.018208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:51.018168    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:55.444688 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:32:55.444648    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:32:55.604430 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:55.604343    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:56.017752 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:56.017716    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:56.059208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:56.059169    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"]
Oct 30 03:32:56.059402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:56.059378    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy" containerID="cri-o://316cdf4f9842f88ac44c64d13befdf913df0f6172900c3f20202c3aca85be10e" gracePeriod=30
Oct 30 03:32:56.183251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:32:56.183217    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:01.235289 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.235254    5061 generic.go:356] "Generic (PLEG): container finished" podID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerID="316cdf4f9842f88ac44c64d13befdf913df0f6172900c3f20202c3aca85be10e" exitCode=0
Oct 30 03:33:01.235661 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.235360    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" event={"ID":"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9","Type":"ContainerDied","Data":"316cdf4f9842f88ac44c64d13befdf913df0f6172900c3f20202c3aca85be10e"}
Oct 30 03:33:01.313997 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.313968    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 03:33:01.413951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.413883    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-envoy\") pod \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") "
Oct 30 03:33:01.413951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.413920    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-data\") pod \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") "
Oct 30 03:33:01.413951 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.413943    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-kube-api-access-hlsnr\" (UniqueName: \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-kube-api-access-hlsnr\") pod \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") "
Oct 30 03:33:01.414235 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.413963    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istiod-ca-cert\") pod \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") "
Oct 30 03:33:01.414235 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.414014    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-token\") pod \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") "
Oct 30 03:33:01.414235 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.414047    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-credential-socket\") pod \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") "
Oct 30 03:33:01.414235 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.414100    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-certs\") pod \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") "
Oct 30 03:33:01.414235 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.414197    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-data" pod "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" (UID: "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:33:01.414464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.414371    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-credential-socket" pod "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" (UID: "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:33:01.414464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.414436    5061 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-data\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.414554 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.414456    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istiod-ca-cert" pod "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" (UID: "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:33:01.414673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.414647    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-certs" pod "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" (UID: "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:33:01.416145 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.416123    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-kube-api-access-hlsnr" pod "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" (UID: "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9"). InnerVolumeSpecName "kube-api-access-hlsnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:33:01.416253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.416212    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-envoy" pod "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" (UID: "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:33:01.416253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.416232    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-token" pod "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" (UID: "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:33:01.514918 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.514873    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-socket\") pod \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") "
Oct 30 03:33:01.515117 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.515001    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-podinfo\") pod \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\" (UID: \"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9\") "
Oct 30 03:33:01.515190 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.515161    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-socket" pod "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" (UID: "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:33:01.515230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.515175    5061 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-token\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.515230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.515217    5061 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-credential-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.515300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.515228    5061 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.515300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.515241    5061 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-envoy\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.515300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.515250    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlsnr\" (UniqueName: \"kubernetes.io/projected/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-kube-api-access-hlsnr\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.515300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.515258    5061 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istiod-ca-cert\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.517253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.517221    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-podinfo" pod "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" (UID: "d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:33:01.615659 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.615625    5061 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-istio-podinfo\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:33:01.615659 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:01.615655    5061 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9-workload-socket\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:33:02.242099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:02.242056    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r" event={"ID":"d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9","Type":"ContainerDied","Data":"e07e96177b7bcd11112a26d97b8f1c0f1b2d81f3e456e7fc1d6f2f23c394ee20"}
Oct 30 03:33:02.242575 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:02.242121    5061 scope.go:122] "RemoveContainer" containerID="316cdf4f9842f88ac44c64d13befdf913df0f6172900c3f20202c3aca85be10e"
Oct 30 03:33:02.242575 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:02.242068    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"
Oct 30 03:33:02.298354 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:02.298323    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"]
Oct 30 03:33:02.309074 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:02.309045    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-aeff1d-openshift-default-5bcf8b7867-9kz2r"]
Oct 30 03:33:02.726425 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:02.726388    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:02.845303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:02.845268    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" path="/var/lib/kubelet/pods/d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9/volumes"
Oct 30 03:33:02.990736 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:02.990632    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:03.169204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:03.169171    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:03.296930 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:03.296894    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:03.868223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:03.868187    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:09.320044 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:09.320005    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:20.166451 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166415    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"]
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166766    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d1a22ae9-6252-4363-b95b-fa3708998135" containerName="maas-api"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166778    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a22ae9-6252-4363-b95b-fa3708998135" containerName="maas-api"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166787    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2a7fd568-5977-42fc-987d-61af621d4d38" containerName="istio-proxy"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166791    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7fd568-5977-42fc-987d-61af621d4d38" containerName="istio-proxy"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166801    5061 cpu_manager.go:414] "containerMap: removing container" podUID="bb0624f0-1f31-469a-84b5-97b1a9c42a8e" containerName="payload-pre-processing"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166823    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0624f0-1f31-469a-84b5-97b1a9c42a8e" containerName="payload-pre-processing"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166836    5061 cpu_manager.go:414] "containerMap: removing container" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166841    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166864    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d7a6eb60-0637-4d7c-9ab3-769ceb0cc464" containerName="payload-processing"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166872    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a6eb60-0637-4d7c-9ab3-769ceb0cc464" containerName="payload-processing"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166898    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerName="istio-proxy"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166906    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerName="istio-proxy"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166917    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerName="cleanup"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166922    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerName="cleanup"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166929    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d29cfa43-cc24-4c84-a55d-12401012d345" containerName="payload-pre-processing"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166934    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d29cfa43-cc24-4c84-a55d-12401012d345" containerName="payload-pre-processing"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166941    5061 cpu_manager.go:414] "containerMap: removing container" podUID="b08d7a98-4183-4f36-918d-3a5df4aaa0ab" containerName="payload-pre-processing"
Oct 30 03:33:20.166934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166946    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08d7a98-4183-4f36-918d-3a5df4aaa0ab" containerName="payload-pre-processing"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166955    5061 cpu_manager.go:414] "containerMap: removing container" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166964    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166970    5061 cpu_manager.go:414] "containerMap: removing container" podUID="70a5c3d4-194d-4214-89bc-697164722e46" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166975    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a5c3d4-194d-4214-89bc-697164722e46" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166983    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166987    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166993    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e3a5a139-8b79-485f-9392-4fd1b0f18561" containerName="payload-processing"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.166998    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a5a139-8b79-485f-9392-4fd1b0f18561" containerName="payload-processing"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167004    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167008    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167017    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e28f99f1-cc49-4c8b-8a34-7440b8068712" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167022    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28f99f1-cc49-4c8b-8a34-7440b8068712" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167030    5061 cpu_manager.go:414] "containerMap: removing container" podUID="70a5c3d4-194d-4214-89bc-697164722e46" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167037    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a5c3d4-194d-4214-89bc-697164722e46" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167046    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167053    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167061    5061 cpu_manager.go:414] "containerMap: removing container" podUID="c2fedcd9-ed61-4532-ae59-8583c533412e" containerName="maas-api"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167068    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fedcd9-ed61-4532-ae59-8583c533412e" containerName="maas-api"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167080    5061 cpu_manager.go:414] "containerMap: removing container" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167087    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167096    5061 cpu_manager.go:414] "containerMap: removing container" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167103    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167115    5061 cpu_manager.go:414] "containerMap: removing container" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167122    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167132    5061 cpu_manager.go:414] "containerMap: removing container" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167139    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167148    5061 cpu_manager.go:414] "containerMap: removing container" podUID="70a5c3d4-194d-4214-89bc-697164722e46" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167155    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a5c3d4-194d-4214-89bc-697164722e46" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167166    5061 cpu_manager.go:414] "containerMap: removing container" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167170    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167182    5061 cpu_manager.go:414] "containerMap: removing container" podUID="338661f3-a26f-486a-a63e-e7b10c7499e2" containerName="payload-processing"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167189    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338661f3-a26f-486a-a63e-e7b10c7499e2" containerName="payload-processing"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167208    5061 cpu_manager.go:414] "containerMap: removing container" podUID="62ce2051-8166-4b9c-94f2-2711be84b2d1" containerName="revoke-keys"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167215    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ce2051-8166-4b9c-94f2-2711be84b2d1" containerName="revoke-keys"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167324    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167338    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a5c3d4-194d-4214-89bc-697164722e46" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167348    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6e7199-66a6-4043-9bdd-ca245968dfef" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167356    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a5a139-8b79-485f-9392-4fd1b0f18561" containerName="payload-processing"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167367    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167378    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28f99f1-cc49-4c8b-8a34-7440b8068712" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167386    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a5c3d4-194d-4214-89bc-697164722e46" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167393    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede5a7fc-13b5-40ea-a716-7b0acdaacc93" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167402    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fedcd9-ed61-4532-ae59-8583c533412e" containerName="maas-api"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167410    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1227d4b-32b5-4ec8-9037-0b830b4528c6" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167417    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="491108be-282e-45ef-ac58-a9404fc21748" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167426    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7f6efd-1f7d-4d89-b020-0e6fb6d077bb" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167432    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec889f00-32b6-4c2c-931e-f42c26a0026c" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167440    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c45f15-d4bc-4f69-9d48-5ccdebcee7a9" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167449    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="338661f3-a26f-486a-a63e-e7b10c7499e2" containerName="payload-processing"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167457    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerName="cleanup"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167465    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ce2051-8166-4b9c-94f2-2711be84b2d1" containerName="revoke-keys"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167474    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a22ae9-6252-4363-b95b-fa3708998135" containerName="maas-api"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167483    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7fd568-5977-42fc-987d-61af621d4d38" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167492    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0624f0-1f31-469a-84b5-97b1a9c42a8e" containerName="payload-pre-processing"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167502    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c837855-e677-46af-91b9-2f2a06306d5b" containerName="main"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167509    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a6eb60-0637-4d7c-9ab3-769ceb0cc464" containerName="payload-processing"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167519    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ce09f3-daae-4d0f-ae5c-41dd7f88fc2f" containerName="istio-proxy"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167529    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="d29cfa43-cc24-4c84-a55d-12401012d345" containerName="payload-pre-processing"
Oct 30 03:33:20.167940 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.167538    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08d7a98-4183-4f36-918d-3a5df4aaa0ab" containerName="payload-pre-processing"
Oct 30 03:33:20.172257 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.172238    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.198766 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.198736    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"]
Oct 30 03:33:20.294010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.293968    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.294010 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.294017    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxkx7\" (UniqueName: \"kubernetes.io/projected/445e1027-8886-446c-87b5-a8e4ed440a06-kube-api-access-nxkx7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.294208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.294112    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.294208 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.294182    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/445e1027-8886-446c-87b5-a8e4ed440a06-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.294280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.294209    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.294280 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.294232    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/445e1027-8886-446c-87b5-a8e4ed440a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.294347 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.294292    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.294380 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.294368    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.294415 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.294401    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/445e1027-8886-446c-87b5-a8e4ed440a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.394985    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/445e1027-8886-446c-87b5-a8e4ed440a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395161 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395041    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395161 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395060    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkx7\" (UniqueName: \"kubernetes.io/projected/445e1027-8886-446c-87b5-a8e4ed440a06-kube-api-access-nxkx7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395161 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395089    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395210    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/445e1027-8886-446c-87b5-a8e4ed440a06-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395265    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395291    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/445e1027-8886-446c-87b5-a8e4ed440a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395326    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395374    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395654 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395606    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395751    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395933    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.395992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.395953    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/445e1027-8886-446c-87b5-a8e4ed440a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.396186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.396163    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.397954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.397930    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/445e1027-8886-446c-87b5-a8e4ed440a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.398042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.397937    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/445e1027-8886-446c-87b5-a8e4ed440a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.412954 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.412929    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkx7\" (UniqueName: \"kubernetes.io/projected/445e1027-8886-446c-87b5-a8e4ed440a06-kube-api-access-nxkx7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.413065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.412977    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/445e1027-8886-446c-87b5-a8e4ed440a06-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-4nnf7\" (UID: \"445e1027-8886-446c-87b5-a8e4ed440a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.484684 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.484620    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:20.793734 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.793707    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"]
Oct 30 03:33:20.798503 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:33:20.798473    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445e1027_8886_446c_87b5_a8e4ed440a06.slice/crio-ed9675d562844d4b278746955d946d5b9177537dd1494b33ca86c5bdf00b2f22 WatchSource:0}: Error finding container ed9675d562844d4b278746955d946d5b9177537dd1494b33ca86c5bdf00b2f22: Status 404 returned error can't find the container with id ed9675d562844d4b278746955d946d5b9177537dd1494b33ca86c5bdf00b2f22
Oct 30 03:33:20.800644 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.800615    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:33:20.800712 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.800679    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:33:20.800751 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:20.800710    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:33:21.350137 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:21.350094    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" event={"ID":"445e1027-8886-446c-87b5-a8e4ed440a06","Type":"ContainerStarted","Data":"50833eb74f5492ab1fdfee914a3dd6145e3f89ad945102a69d928340c1e0d95b"}
Oct 30 03:33:21.350665 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:21.350193    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" event={"ID":"445e1027-8886-446c-87b5-a8e4ed440a06","Type":"ContainerStarted","Data":"ed9675d562844d4b278746955d946d5b9177537dd1494b33ca86c5bdf00b2f22"}
Oct 30 03:33:21.409430 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:21.409385    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" podStartSLOduration=1.409372437 podStartE2EDuration="1.409372437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:21.397513389 +0000 UTC m=+3146.699691770" watchObservedRunningTime="2026-10-30 03:33:21.409372437 +0000 UTC m=+3146.711550871"
Oct 30 03:33:21.485219 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:21.485174    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:21.486821 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:21.486794    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:33:21.486974 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:21.486867    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" podUID="445e1027-8886-446c-87b5-a8e4ed440a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 03:33:22.485984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:22.485951    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:33:22.486471 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:22.486002    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" podUID="445e1027-8886-446c-87b5-a8e4ed440a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 03:33:23.485758 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:23.485723    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:33:23.485985 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:23.485782    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" podUID="445e1027-8886-446c-87b5-a8e4ed440a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 03:33:24.485619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:24.485585    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:33:24.486042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:24.485649    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" podUID="445e1027-8886-446c-87b5-a8e4ed440a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 03:33:25.465347 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:33:25.465308    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:33:25.485902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:25.485875    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:33:25.486238 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:25.485926    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" podUID="445e1027-8886-446c-87b5-a8e4ed440a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 03:33:26.485684 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:26.485653    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:33:26.485915 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:26.485717    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" podUID="445e1027-8886-446c-87b5-a8e4ed440a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 03:33:27.485751 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:27.485709    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:33:27.485957 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:27.485778    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" podUID="445e1027-8886-446c-87b5-a8e4ed440a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 03:33:29.486784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:29.486745    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:33:29.487350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:29.487310    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" podUID="445e1027-8886-446c-87b5-a8e4ed440a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:33:30.527993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:30.527956    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:33:30.528518 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:30.528010    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" podUID="445e1027-8886-446c-87b5-a8e4ed440a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:33:31.082384 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:31.082353    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:31.082665 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:31.082630    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:31.083358 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:31.083339    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7"
Oct 30 03:33:31.968131 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:31.968094    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:34.544025 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:34.543983    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:35.207352 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.207319    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"]
Oct 30 03:33:35.207768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.207751    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerName="cleanup"
Oct 30 03:33:35.207768 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.207768    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerName="cleanup"
Oct 30 03:33:35.208127 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.207912    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a5c3d4-194d-4214-89bc-697164722e46" containerName="cleanup"
Oct 30 03:33:35.208127 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.207928    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b27eb1a-516b-4ec1-b5b4-afc8333f6984" containerName="cleanup"
Oct 30 03:33:35.211267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.211244    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.240890 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.240860    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"]
Oct 30 03:33:35.252904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.252877    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.253021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.252917    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdk6\" (UniqueName: \"kubernetes.io/projected/6f86e91a-cb8a-41cc-8c16-879a462fea69-kube-api-access-ptdk6\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.253021 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.253004    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.253125 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.253090    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.253125 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.253114    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.253223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.253143    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.253342 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.253319    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f86e91a-cb8a-41cc-8c16-879a462fea69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.253410 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.253372    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.253481 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.253452    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.354658 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.354627    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.354803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.354672    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.354803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.354705    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdk6\" (UniqueName: \"kubernetes.io/projected/6f86e91a-cb8a-41cc-8c16-879a462fea69-kube-api-access-ptdk6\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.354803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.354734    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.354803 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.354771    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.355065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.354811    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.355065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.354876    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.355065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.354966    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f86e91a-cb8a-41cc-8c16-879a462fea69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.355065 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.354997    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.355309 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.355182    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.355369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.355309    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.355412 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.355363    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.355464 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.355447    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.355686 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.355667    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f86e91a-cb8a-41cc-8c16-879a462fea69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.357547 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.357521    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.357657 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.357641    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.372559 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.372532    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f86e91a-cb8a-41cc-8c16-879a462fea69-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.372764 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.372747    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdk6\" (UniqueName: \"kubernetes.io/projected/6f86e91a-cb8a-41cc-8c16-879a462fea69-kube-api-access-ptdk6\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pnfx8\" (UID: \"6f86e91a-cb8a-41cc-8c16-879a462fea69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.525077 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.524978    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:35.836272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.836235    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"]
Oct 30 03:33:35.840500 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:33:35.840466    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f86e91a_cb8a_41cc_8c16_879a462fea69.slice/crio-3496a2c9193a9bf9f1260dce0ab145cf9c67a1168d8ad0ba9f04a2723af53215 WatchSource:0}: Error finding container 3496a2c9193a9bf9f1260dce0ab145cf9c67a1168d8ad0ba9f04a2723af53215: Status 404 returned error can't find the container with id 3496a2c9193a9bf9f1260dce0ab145cf9c67a1168d8ad0ba9f04a2723af53215
Oct 30 03:33:35.842608 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.842576    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:33:35.842711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.842652    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:33:35.842711 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:35.842696    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:33:36.439870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:36.433687    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8" event={"ID":"6f86e91a-cb8a-41cc-8c16-879a462fea69","Type":"ContainerStarted","Data":"e358b37d952e4aae91d32fc7427271c364eaad02f5b8fe8b085148be97e500c5"}
Oct 30 03:33:36.439870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:36.433761    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8" event={"ID":"6f86e91a-cb8a-41cc-8c16-879a462fea69","Type":"ContainerStarted","Data":"3496a2c9193a9bf9f1260dce0ab145cf9c67a1168d8ad0ba9f04a2723af53215"}
Oct 30 03:33:36.481978 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:36.481923    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8" podStartSLOduration=1.481909275 podStartE2EDuration="1.481909275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:36.480972951 +0000 UTC m=+3161.783151316" watchObservedRunningTime="2026-10-30 03:33:36.481909275 +0000 UTC m=+3161.784087640"
Oct 30 03:33:36.527778 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:36.527734    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:33:36.528176 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:36.528144    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8" podUID="6f86e91a-cb8a-41cc-8c16-879a462fea69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 03:33:36.528748 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:36.525632    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:37.538310 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:37.538275    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:33:37.538928 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:37.538895    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8" podUID="6f86e91a-cb8a-41cc-8c16-879a462fea69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 03:33:38.539872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:38.535170    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:33:38.539872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:38.535227    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8" podUID="6f86e91a-cb8a-41cc-8c16-879a462fea69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 03:33:39.536871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:39.530188    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:33:39.536871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:39.530250    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8" podUID="6f86e91a-cb8a-41cc-8c16-879a462fea69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 03:33:40.527265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:40.527227    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:33:40.527904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:40.527842    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8" podUID="6f86e91a-cb8a-41cc-8c16-879a462fea69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 03:33:41.540289 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:41.540245    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:33:41.540941 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:41.540906    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8" podUID="6f86e91a-cb8a-41cc-8c16-879a462fea69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 03:33:42.525569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:42.525529    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:33:42.525808 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:42.525585    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8" podUID="6f86e91a-cb8a-41cc-8c16-879a462fea69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 03:33:44.077474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:44.077436    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:44.077985 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:44.077831    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:44.078574 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:44.078555    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8"
Oct 30 03:33:51.136108 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:33:51.136073    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:55.465123 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:33:55.465036    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:34:25.445099 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:34:25.445059    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:34:35.460268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.460228    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"]
Oct 30 03:34:35.464128 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.464106    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.490526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.490494    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"]
Oct 30 03:34:35.554352 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.554316    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.554499 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.554369    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.554499 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.554480    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.554593 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.554515    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.554593 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.554540    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.554666 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.554625    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.554703 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.554693    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.554738 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.554721    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.554776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.554736    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxjng\" (UniqueName: \"kubernetes.io/projected/f65ce0e1-cada-4a84-a7b7-1da0a01595da-kube-api-access-mxjng\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656185 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656157    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656347 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656198    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656347 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656221    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656347 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656263    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656347 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656290    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656347 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656317    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656440    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656483    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxjng\" (UniqueName: \"kubernetes.io/projected/f65ce0e1-cada-4a84-a7b7-1da0a01595da-kube-api-access-mxjng\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656537    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656702    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656815    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.656976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.656913    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.657036 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.657021    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.657390 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.657362    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.658643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.658624    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.658770 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.658753    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.673617 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.673598    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f65ce0e1-cada-4a84-a7b7-1da0a01595da-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.674017 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.674000    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxjng\" (UniqueName: \"kubernetes.io/projected/f65ce0e1-cada-4a84-a7b7-1da0a01595da-kube-api-access-mxjng\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4p7\" (UID: \"f65ce0e1-cada-4a84-a7b7-1da0a01595da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:35.778040 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:35.777978    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:36.095363 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:36.095284    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"]
Oct 30 03:34:36.099046 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:34:36.098971    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65ce0e1_cada_4a84_a7b7_1da0a01595da.slice/crio-7e18d4b8f44079793d66a34584f6bbd2c712aaa43c76cc4da80c8ff8f54c13fb WatchSource:0}: Error finding container 7e18d4b8f44079793d66a34584f6bbd2c712aaa43c76cc4da80c8ff8f54c13fb: Status 404 returned error can't find the container with id 7e18d4b8f44079793d66a34584f6bbd2c712aaa43c76cc4da80c8ff8f54c13fb
Oct 30 03:34:36.101245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:36.101215    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:36.101357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:36.101290    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:36.101357 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:36.101335    5061 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:36.747804 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:36.747765    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" event={"ID":"f65ce0e1-cada-4a84-a7b7-1da0a01595da","Type":"ContainerStarted","Data":"c332c60dc90a763343bba4b5ceca5c7e7f3c2bd21ad32a3c6a540c8a4a032514"}
Oct 30 03:34:36.748277 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:36.747835    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" event={"ID":"f65ce0e1-cada-4a84-a7b7-1da0a01595da","Type":"ContainerStarted","Data":"7e18d4b8f44079793d66a34584f6bbd2c712aaa43c76cc4da80c8ff8f54c13fb"}
Oct 30 03:34:36.778829 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:36.778797    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:36.799292 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:36.799245    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" podStartSLOduration=1.799232014 podStartE2EDuration="1.799232014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:36.798102531 +0000 UTC m=+3222.100280896" watchObservedRunningTime="2026-10-30 03:34:36.799232014 +0000 UTC m=+3222.101410380"
Oct 30 03:34:37.780600 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:37.780570    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 30 03:34:37.781023 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:37.780618    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" podUID="f65ce0e1-cada-4a84-a7b7-1da0a01595da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 30 03:34:38.778904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:38.778871    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 30 03:34:38.779087 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:38.778933    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" podUID="f65ce0e1-cada-4a84-a7b7-1da0a01595da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 30 03:34:39.779003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:39.778973    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 30 03:34:39.779422 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:39.779032    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" podUID="f65ce0e1-cada-4a84-a7b7-1da0a01595da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 30 03:34:40.778569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:40.778535    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 30 03:34:40.778759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:40.778587    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" podUID="f65ce0e1-cada-4a84-a7b7-1da0a01595da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 30 03:34:41.778583 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:41.778542    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 30 03:34:41.779006 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:41.778607    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" podUID="f65ce0e1-cada-4a84-a7b7-1da0a01595da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 30 03:34:42.779284 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:42.779249    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 30 03:34:42.779695 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:42.779304    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" podUID="f65ce0e1-cada-4a84-a7b7-1da0a01595da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 30 03:34:44.736674 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:44.736635    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:44.778585 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:44.778548    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:34:44.778731 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:44.778610    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" podUID="f65ce0e1-cada-4a84-a7b7-1da0a01595da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:34:45.820030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:45.819991    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:34:45.820533 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:45.820101    5061 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" podUID="f65ce0e1-cada-4a84-a7b7-1da0a01595da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:34:46.275353 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:46.275320    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:46.275628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:46.275601    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:46.276249 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:46.276232    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7"
Oct 30 03:34:51.225945 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:34:51.225913    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:55.445565 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:34:55.445527    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:35:10.835124 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:35:10.835090    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:10.960435 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:35:10.960402    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:24.554871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:35:24.553976    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-54ldl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:35:24.554871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:35:24.554088    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" podUID="91dfc434-1255-4315-a28d-b85d3f649964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:35:25.444974 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:35:25.444939    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:35:31.682303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:35:31.682259    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:51.434552 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:35:51.434509    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:55.446506 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:35:55.446469    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:35:55.497159 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:35:55.497126    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:35:55.500137 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:35:55.500111    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:35:55.501929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:35:55.501904    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:35:55.504871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:35:55.504828    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:36:25.446186 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:36:25.446147    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:36:51.438505 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:36:51.438464    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:55.444983 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:36:55.444942    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:36:57.622977 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:36:57.622944    5061 scope.go:122] "RemoveContainer" containerID="497da6529e9b2075717d02457893f3dca7206e07392e2f5500293dd62e869213"
Oct 30 03:37:25.445699 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:37:25.445657    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:37:51.580780 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:51.580747    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:51.701619 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:51.701577    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:52.096763 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:52.096726    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:52.280549 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:52.280517    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:55.445895 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:37:55.445838    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:37:55.559422 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:55.559384    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:55.698761 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:55.698679    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:57.151316 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.151281    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"]
Oct 30 03:37:57.155006 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.154985    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.168472 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.168451    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ed4e18-serving-cert\""
Oct 30 03:37:57.175031 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.175006    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"]
Oct 30 03:37:57.180582 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.180558    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.180753 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.180727    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829da91a-ed69-47a3-a18f-74d8c27c4c61-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.180900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.180817    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.180900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.180891    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtc6q\" (UniqueName: \"kubernetes.io/projected/829da91a-ed69-47a3-a18f-74d8c27c4c61-kube-api-access-dtc6q\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.268673 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.268635    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:57.282311 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.282277    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.282495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.282315    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829da91a-ed69-47a3-a18f-74d8c27c4c61-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.282495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.282359    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.282495 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.282391    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtc6q\" (UniqueName: \"kubernetes.io/projected/829da91a-ed69-47a3-a18f-74d8c27c4c61-kube-api-access-dtc6q\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.282685 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:37:57.282586    5061 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-serving-cert: secret "maas-api-e2e-worker-single-main-ed4e18-serving-cert" not found
Oct 30 03:37:57.282685 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:37:57.282662    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-maas-api-tls podName:829da91a-ed69-47a3-a18f-74d8c27c4c61 nodeName:}" failed. No retries permitted until 2026-10-30 03:37:57.782634113 +0000 UTC m=+3423.084812467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-maas-api-tls") pod "maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8" (UID: "829da91a-ed69-47a3-a18f-74d8c27c4c61") : secret "maas-api-e2e-worker-single-main-ed4e18-serving-cert" not found
Oct 30 03:37:57.283310 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.283284    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829da91a-ed69-47a3-a18f-74d8c27c4c61-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.284959 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.284928    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.302114 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.302090    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtc6q\" (UniqueName: \"kubernetes.io/projected/829da91a-ed69-47a3-a18f-74d8c27c4c61-kube-api-access-dtc6q\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.389057 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.389015    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:57.451592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.451527    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"]
Oct 30 03:37:57.455228 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.455203    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:37:57.473653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.473539    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ed4e18\""
Oct 30 03:37:57.475622 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.475602    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"]
Oct 30 03:37:57.484017 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.483993    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8428bdda-18a6-4826-8978-b7f888c9e422-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs\" (UID: \"8428bdda-18a6-4826-8978-b7f888c9e422\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:37:57.484132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.484035    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlx29\" (UniqueName: \"kubernetes.io/projected/8428bdda-18a6-4826-8978-b7f888c9e422-kube-api-access-zlx29\") pod \"payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs\" (UID: \"8428bdda-18a6-4826-8978-b7f888c9e422\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:37:57.519654 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.519616    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:57.542976 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.542295    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ffe9d7be-dockercfg-gwcqx\""
Oct 30 03:37:57.585251 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.585204    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8428bdda-18a6-4826-8978-b7f888c9e422-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs\" (UID: \"8428bdda-18a6-4826-8978-b7f888c9e422\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:37:57.585446 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.585267    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlx29\" (UniqueName: \"kubernetes.io/projected/8428bdda-18a6-4826-8978-b7f888c9e422-kube-api-access-zlx29\") pod \"payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs\" (UID: \"8428bdda-18a6-4826-8978-b7f888c9e422\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:37:57.586131 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.586102    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8428bdda-18a6-4826-8978-b7f888c9e422-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs\" (UID: \"8428bdda-18a6-4826-8978-b7f888c9e422\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:37:57.602983 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.602958    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlx29\" (UniqueName: \"kubernetes.io/projected/8428bdda-18a6-4826-8978-b7f888c9e422-kube-api-access-zlx29\") pod \"payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs\" (UID: \"8428bdda-18a6-4826-8978-b7f888c9e422\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:37:57.769034 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.768963    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:37:57.788091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.788056    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:57.790689 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:57.790666    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:58.066213 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.066180    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"]
Oct 30 03:37:58.068409 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.068383    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:58.070832 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:37:58.070805    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8428bdda_18a6_4826_8978_b7f888c9e422.slice/crio-4855510918ba7a876efb8d128c72fbc13c62e9b2ad1da6466a8e24bea6668d0c WatchSource:0}: Error finding container 4855510918ba7a876efb8d128c72fbc13c62e9b2ad1da6466a8e24bea6668d0c: Status 404 returned error can't find the container with id 4855510918ba7a876efb8d128c72fbc13c62e9b2ad1da6466a8e24bea6668d0c
Oct 30 03:37:58.371352 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.371275    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"]
Oct 30 03:37:58.375857 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:37:58.375820    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829da91a_ed69_47a3_a18f_74d8c27c4c61.slice/crio-da74d40ced4a7f1713c412e426ca449467ec2f0956c01ee405fb44631da3784c WatchSource:0}: Error finding container da74d40ced4a7f1713c412e426ca449467ec2f0956c01ee405fb44631da3784c: Status 404 returned error can't find the container with id da74d40ced4a7f1713c412e426ca449467ec2f0956c01ee405fb44631da3784c
Oct 30 03:37:58.377541 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.377516    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:37:58.920644 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.920611    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8" event={"ID":"829da91a-ed69-47a3-a18f-74d8c27c4c61","Type":"ContainerStarted","Data":"f0d62e3b320ccda56d2243a5198c7ac2a806a45d1c4e315d6ebaf6fe85a71573"}
Oct 30 03:37:58.920838 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.920670    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8" event={"ID":"829da91a-ed69-47a3-a18f-74d8c27c4c61","Type":"ContainerStarted","Data":"da74d40ced4a7f1713c412e426ca449467ec2f0956c01ee405fb44631da3784c"}
Oct 30 03:37:58.920838 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.920716    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:37:58.922165 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.922141    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs" event={"ID":"8428bdda-18a6-4826-8978-b7f888c9e422","Type":"ContainerStarted","Data":"8d59861b05c81a9f1920ecf77fe3812f00176473ddcd0622f1bbc223f82a1872"}
Oct 30 03:37:58.922272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.922177    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs" event={"ID":"8428bdda-18a6-4826-8978-b7f888c9e422","Type":"ContainerStarted","Data":"4855510918ba7a876efb8d128c72fbc13c62e9b2ad1da6466a8e24bea6668d0c"}
Oct 30 03:37:58.922272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.922257    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:37:58.957754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.957705    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8" podStartSLOduration=1.5455848460000001 podStartE2EDuration="1.957694861s" totalImagesPullingTime="412.110015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:58.956253936 +0000 UTC m=+3424.258432303" watchObservedRunningTime="2026-10-30 03:37:58.957694861 +0000 UTC m=+3424.259873223"
Oct 30 03:37:58.990330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:58.990290    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs" podStartSLOduration=1.990278674 podStartE2EDuration="1.990278674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:58.989944707 +0000 UTC m=+3424.292123083" watchObservedRunningTime="2026-10-30 03:37:58.990278674 +0000 UTC m=+3424.292457038"
Oct 30 03:37:59.103813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:37:59.103778    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:00.723063 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:00.723029    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:02.699091 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:02.699058    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:04.495267 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:04.495231    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:04.932555 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:04.932527    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:38:06.374303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:06.374266    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:08.188957 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:08.188922    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:09.858020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:09.857984    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:09.929590 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:09.929564    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:38:10.601266 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:10.601231    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:11.263539 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:11.263508    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:11.399155 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:11.399118    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:11.522403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:11.522319    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:11.724773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:11.724742    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:13.461676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:13.461640    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:13.921456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:13.921397    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"]
Oct 30 03:38:13.925721 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:13.925694    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:13.939468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:13.939442    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ed4e18\"/\"default-dockercfg-gzllx\""
Oct 30 03:38:13.950476 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:13.950447    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ed4e18\"/\"simulator-single-main-ed4e18-kserve-self-signed-certs\""
Oct 30 03:38:13.950626 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:13.950513    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ed4e18\"/\"openshift-service-ca.crt\""
Oct 30 03:38:13.952645 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:13.952620    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ed4e18\"/\"kube-root-ca.crt\""
Oct 30 03:38:13.952772 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:13.952665    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"]
Oct 30 03:38:14.008360 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.008308    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:14.042581 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.042551    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbw76\" (UniqueName: \"kubernetes.io/projected/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-kube-api-access-bbw76\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.042696 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.042593    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-model-cache\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.042746 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.042712    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-dshm\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.042785 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.042754    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tmp-dir\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.042822 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.042803    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-home\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.042884 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.042875    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tls-certs\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.144099 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.144073    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-dshm\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.144238 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.144106    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tmp-dir\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.144238 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.144159    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-home\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.144238 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.144183    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tls-certs\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.144238 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.144220    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbw76\" (UniqueName: \"kubernetes.io/projected/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-kube-api-access-bbw76\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.144474 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.144260    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-model-cache\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.144598 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.144572    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tmp-dir\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.144651 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.144635    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-home\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.144687 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.144650    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-model-cache\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.146445 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.146419    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-dshm\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.146704 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.146689    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tls-certs\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.162326 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.162296    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbw76\" (UniqueName: \"kubernetes.io/projected/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-kube-api-access-bbw76\") pod \"simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") " pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.240395 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.240311    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:14.533928 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.533856    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"]
Oct 30 03:38:14.538590 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:38:14.538564    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafb90ae7_4344_49d7_ae1b_3d8a8a1540a0.slice/crio-0e909e7623d6d300ecf745a974252f4201f848d0bc719706888d4ad8ddbc5f7a WatchSource:0}: Error finding container 0e909e7623d6d300ecf745a974252f4201f848d0bc719706888d4ad8ddbc5f7a: Status 404 returned error can't find the container with id 0e909e7623d6d300ecf745a974252f4201f848d0bc719706888d4ad8ddbc5f7a
Oct 30 03:38:14.995747 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.995705    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62" event={"ID":"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0","Type":"ContainerStarted","Data":"5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74"}
Oct 30 03:38:14.995975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:14.995760    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62" event={"ID":"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0","Type":"ContainerStarted","Data":"0e909e7623d6d300ecf745a974252f4201f848d0bc719706888d4ad8ddbc5f7a"}
Oct 30 03:38:15.032526 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:15.032427    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62" podStartSLOduration=2.032407376 podStartE2EDuration="2.032407376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:38:15.030374762 +0000 UTC m=+3440.332553137" watchObservedRunningTime="2026-10-30 03:38:15.032407376 +0000 UTC m=+3440.334585742"
Oct 30 03:38:15.241169 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:15.241131    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:15.313957 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:15.313902    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:17.056379 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:17.056344    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:18.852431 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:18.852394    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:20.544095 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:20.544055    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:22.507067 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:22.507029    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:24.168160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:24.168125    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:24.241264 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:24.241226    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:24.254324 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:24.254287    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:25.059643 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:25.059553    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:38:25.468466 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:38:25.468221    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:38:25.808304 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:25.808205    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:27.606043 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:27.605981    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:29.234791 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:29.234754    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:31.206904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:31.206869    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:32.884576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:32.884542    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:34.494009 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:34.493971    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:36.171020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:36.170929    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:37.903157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:37.903124    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:39.627312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:39.627281    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:41.327656 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:41.327618    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:42.995631 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:42.995590    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:44.594139 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:44.594100    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:46.511499 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:46.511462    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:47.492893 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:47.492859    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:48.306648 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:48.306613    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:50.164483 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:50.164449    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:51.687900 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:51.687866    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:51.855162 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:51.855123    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:52.434371 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:52.434331    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"]
Oct 30 03:38:52.525628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:52.525592    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:53.535616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:53.535570    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:55.187312 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:55.187273    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:55.445237 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:38:55.445154    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:38:56.787688 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:56.787653    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:58.112511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:58.112472    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"]
Oct 30 03:38:58.236571 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:58.236538    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:58.349176 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:58.349143    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:58.501630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:58.501556    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:59.804975 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:59.804940    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"]
Oct 30 03:38:59.805508 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:59.805183    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8" podUID="829da91a-ed69-47a3-a18f-74d8c27c4c61" containerName="maas-api" containerID="cri-o://f0d62e3b320ccda56d2243a5198c7ac2a806a45d1c4e315d6ebaf6fe85a71573" gracePeriod=30
Oct 30 03:38:59.883645 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:59.883608    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:59.926919 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:38:59.926881    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8" podUID="829da91a-ed69-47a3-a18f-74d8c27c4c61" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.191:8443/health\": dial tcp 10.132.0.191:8443: connect: connection refused"
Oct 30 03:39:00.221171 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.221132    5061 generic.go:356] "Generic (PLEG): container finished" podID="829da91a-ed69-47a3-a18f-74d8c27c4c61" containerID="f0d62e3b320ccda56d2243a5198c7ac2a806a45d1c4e315d6ebaf6fe85a71573" exitCode=0
Oct 30 03:39:00.221338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.221278    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8" event={"ID":"829da91a-ed69-47a3-a18f-74d8c27c4c61","Type":"ContainerDied","Data":"f0d62e3b320ccda56d2243a5198c7ac2a806a45d1c4e315d6ebaf6fe85a71573"}
Oct 30 03:39:00.263270 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.263248    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:39:00.387662 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.387584    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:00.417564 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.417535    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-metrics-certs\") pod \"829da91a-ed69-47a3-a18f-74d8c27c4c61\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") "
Oct 30 03:39:00.417718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.417591    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-maas-api-tls\") pod \"829da91a-ed69-47a3-a18f-74d8c27c4c61\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") "
Oct 30 03:39:00.417718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.417625    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829da91a-ed69-47a3-a18f-74d8c27c4c61-kube-api-access-dtc6q\" (UniqueName: \"kubernetes.io/projected/829da91a-ed69-47a3-a18f-74d8c27c4c61-kube-api-access-dtc6q\") pod \"829da91a-ed69-47a3-a18f-74d8c27c4c61\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") "
Oct 30 03:39:00.417718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.417687    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829da91a-ed69-47a3-a18f-74d8c27c4c61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829da91a-ed69-47a3-a18f-74d8c27c4c61-openshift-service-ca\") pod \"829da91a-ed69-47a3-a18f-74d8c27c4c61\" (UID: \"829da91a-ed69-47a3-a18f-74d8c27c4c61\") "
Oct 30 03:39:00.418167 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.418135    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829da91a-ed69-47a3-a18f-74d8c27c4c61-openshift-service-ca" pod "829da91a-ed69-47a3-a18f-74d8c27c4c61" (UID: "829da91a-ed69-47a3-a18f-74d8c27c4c61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:00.419700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.419681    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-maas-api-tls" pod "829da91a-ed69-47a3-a18f-74d8c27c4c61" (UID: "829da91a-ed69-47a3-a18f-74d8c27c4c61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:39:00.419810 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.419786    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-metrics-certs" pod "829da91a-ed69-47a3-a18f-74d8c27c4c61" (UID: "829da91a-ed69-47a3-a18f-74d8c27c4c61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:39:00.419870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.419797    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829da91a-ed69-47a3-a18f-74d8c27c4c61-kube-api-access-dtc6q" pod "829da91a-ed69-47a3-a18f-74d8c27c4c61" (UID: "829da91a-ed69-47a3-a18f-74d8c27c4c61"). InnerVolumeSpecName "kube-api-access-dtc6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:00.505754 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.505726    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:00.519258 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.519229    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:00.519258 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.519256    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtc6q\" (UniqueName: \"kubernetes.io/projected/829da91a-ed69-47a3-a18f-74d8c27c4c61-kube-api-access-dtc6q\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:00.519493 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.519266    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829da91a-ed69-47a3-a18f-74d8c27c4c61-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:00.519493 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:00.519277    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829da91a-ed69-47a3-a18f-74d8c27c4c61-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:01.081699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:01.081666    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:01.233921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:01.233813    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"
Oct 30 03:39:01.233921 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:01.233826    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8" event={"ID":"829da91a-ed69-47a3-a18f-74d8c27c4c61","Type":"ContainerDied","Data":"da74d40ced4a7f1713c412e426ca449467ec2f0956c01ee405fb44631da3784c"}
Oct 30 03:39:01.234122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:01.233928    5061 scope.go:122] "RemoveContainer" containerID="f0d62e3b320ccda56d2243a5198c7ac2a806a45d1c4e315d6ebaf6fe85a71573"
Oct 30 03:39:01.298080 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:01.298014    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"]
Oct 30 03:39:01.308538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:01.308507    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ed4e18-8555844bcd-jlsp8"]
Oct 30 03:39:02.846405 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:02.846372    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829da91a-ed69-47a3-a18f-74d8c27c4c61" path="/var/lib/kubelet/pods/829da91a-ed69-47a3-a18f-74d8c27c4c61/volumes"
Oct 30 03:39:03.113676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:03.113580    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs" podUID="8428bdda-18a6-4826-8978-b7f888c9e422" containerName="payload-pre-processing" containerID="cri-o://8d59861b05c81a9f1920ecf77fe3812f00176473ddcd0622f1bbc223f82a1872" gracePeriod=25
Oct 30 03:39:06.809351 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:06.809316    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:06.930434 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:06.930401    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:07.168188 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.168104    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:07.455009 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.454923    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62" podUID="afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" containerName="main" containerID="cri-o://5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74" gracePeriod=45
Oct 30 03:39:07.707179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.707116    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:39:07.887637 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.887606    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-kube-api-access-bbw76\" (UniqueName: \"kubernetes.io/projected/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-kube-api-access-bbw76\") pod \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") "
Oct 30 03:39:07.888154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.887690    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tls-certs\") pod \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") "
Oct 30 03:39:07.888154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.887750    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-dshm\") pod \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") "
Oct 30 03:39:07.888154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.887793    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-model-cache\") pod \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") "
Oct 30 03:39:07.888154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.887856    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tmp-dir\") pod \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") "
Oct 30 03:39:07.888154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.887898    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-home\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-home\") pod \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\" (UID: \"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0\") "
Oct 30 03:39:07.888154 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.888090    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-model-cache" pod "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" (UID: "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:07.888489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.888214    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-home" pod "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" (UID: "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:07.888489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.888317    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tmp-dir" pod "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" (UID: "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:07.888489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.888327    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:07.888489 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.888366    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:07.890394 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.890356    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-dshm" pod "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" (UID: "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:07.897964 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.897936    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tls-certs" pod "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" (UID: "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:39:07.898072 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.898007    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-kube-api-access-bbw76" pod "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" (UID: "afb90ae7-4344-49d7-ae1b-3d8a8a1540a0"). InnerVolumeSpecName "kube-api-access-bbw76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:07.989160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.989103    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbw76\" (UniqueName: \"kubernetes.io/projected/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-kube-api-access-bbw76\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:07.989160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.989127    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:07.989160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.989137    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:07.989160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:07.989162    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:08.268092 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:08.267994    5061 generic.go:356] "Generic (PLEG): container finished" podID="afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" containerID="5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74" exitCode=0
Oct 30 03:39:08.268092 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:08.268047    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"
Oct 30 03:39:08.268295 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:08.268082    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62" event={"ID":"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0","Type":"ContainerDied","Data":"5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74"}
Oct 30 03:39:08.268295 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:08.268136    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62" event={"ID":"afb90ae7-4344-49d7-ae1b-3d8a8a1540a0","Type":"ContainerDied","Data":"0e909e7623d6d300ecf745a974252f4201f848d0bc719706888d4ad8ddbc5f7a"}
Oct 30 03:39:08.268295 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:08.268153    5061 scope.go:122] "RemoveContainer" containerID="5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74"
Oct 30 03:39:08.278085 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:08.278069    5061 scope.go:122] "RemoveContainer" containerID="5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74"
Oct 30 03:39:08.278373 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:39:08.278351    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74\": container with ID starting with 5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74 not found: ID does not exist" containerID="5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74"
Oct 30 03:39:08.278449 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:08.278381    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74"} err="failed to get container status \"5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74\": rpc error: code = NotFound desc = could not find container \"5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74\": container with ID starting with 5fb8a4426b08476fe3bbee7f791c39207dfa7bc1a9f6dc418a48d79a7aa2ca74 not found: ID does not exist"
Oct 30 03:39:08.312985 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:08.312961    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"]
Oct 30 03:39:08.323528 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:08.323499    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ed4e18/simulator-single-main-ed4e18-kserve-6d8655b5d5-tpt62"]
Oct 30 03:39:08.845881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:08.845833    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" path="/var/lib/kubelet/pods/afb90ae7-4344-49d7-ae1b-3d8a8a1540a0/volumes"
Oct 30 03:39:09.927692 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:09.927661    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs" podUID="8428bdda-18a6-4826-8978-b7f888c9e422" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:39:18.305410 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:18.305361    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:19.928035 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:19.927998    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs" podUID="8428bdda-18a6-4826-8978-b7f888c9e422" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:39:23.336749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:23.336714    5061 generic.go:356] "Generic (PLEG): container finished" podID="8428bdda-18a6-4826-8978-b7f888c9e422" containerID="8d59861b05c81a9f1920ecf77fe3812f00176473ddcd0622f1bbc223f82a1872" exitCode=0
Oct 30 03:39:23.337163 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:23.336794    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs" event={"ID":"8428bdda-18a6-4826-8978-b7f888c9e422","Type":"ContainerDied","Data":"8d59861b05c81a9f1920ecf77fe3812f00176473ddcd0622f1bbc223f82a1872"}
Oct 30 03:39:23.366425 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:23.366405    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:39:23.426349 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:23.426281    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8428bdda-18a6-4826-8978-b7f888c9e422-kube-api-access-zlx29\" (UniqueName: \"kubernetes.io/projected/8428bdda-18a6-4826-8978-b7f888c9e422-kube-api-access-zlx29\") pod \"8428bdda-18a6-4826-8978-b7f888c9e422\" (UID: \"8428bdda-18a6-4826-8978-b7f888c9e422\") "
Oct 30 03:39:23.426468 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:23.426451    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8428bdda-18a6-4826-8978-b7f888c9e422-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8428bdda-18a6-4826-8978-b7f888c9e422-plugins-config-volume\") pod \"8428bdda-18a6-4826-8978-b7f888c9e422\" (UID: \"8428bdda-18a6-4826-8978-b7f888c9e422\") "
Oct 30 03:39:23.426798 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:23.426772    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8428bdda-18a6-4826-8978-b7f888c9e422-plugins-config-volume" pod "8428bdda-18a6-4826-8978-b7f888c9e422" (UID: "8428bdda-18a6-4826-8978-b7f888c9e422"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:23.428392 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:23.428367    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8428bdda-18a6-4826-8978-b7f888c9e422-kube-api-access-zlx29" pod "8428bdda-18a6-4826-8978-b7f888c9e422" (UID: "8428bdda-18a6-4826-8978-b7f888c9e422"). InnerVolumeSpecName "kube-api-access-zlx29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:23.527647 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:23.527612    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8428bdda-18a6-4826-8978-b7f888c9e422-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:23.527647 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:23.527641    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlx29\" (UniqueName: \"kubernetes.io/projected/8428bdda-18a6-4826-8978-b7f888c9e422-kube-api-access-zlx29\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:39:24.345049 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:24.345008    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"
Oct 30 03:39:24.345601 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:24.345046    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs" event={"ID":"8428bdda-18a6-4826-8978-b7f888c9e422","Type":"ContainerDied","Data":"4855510918ba7a876efb8d128c72fbc13c62e9b2ad1da6466a8e24bea6668d0c"}
Oct 30 03:39:24.345601 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:24.345113    5061 scope.go:122] "RemoveContainer" containerID="8d59861b05c81a9f1920ecf77fe3812f00176473ddcd0622f1bbc223f82a1872"
Oct 30 03:39:24.391740 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:24.391699    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"]
Oct 30 03:39:24.402373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:24.402317    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ed4e18-6585dqjwjs"]
Oct 30 03:39:24.845008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:24.844967    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8428bdda-18a6-4826-8978-b7f888c9e422" path="/var/lib/kubelet/pods/8428bdda-18a6-4826-8978-b7f888c9e422/volumes"
Oct 30 03:39:25.445808 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:39:25.445706    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:39:28.656287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:28.656257    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:29.645713 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:29.645679    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:35.286645 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:35.286549    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:51.784416 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:39:51.784383    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:55.444856 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:39:55.444805    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:40:02.223453 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:40:02.223419    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:25.445381 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:40:25.445342    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:40:52.020765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:40:52.020725    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:55.445090 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:40:55.445056    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:40:55.527171 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:40:55.527145    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:40:55.530176 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:40:55.530146    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:40:55.531987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:40:55.531961    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:40:55.534962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:40:55.534943    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:41:09.560869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:09.559941    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-54ldl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:41:09.560869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:09.560022    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" podUID="91dfc434-1255-4315-a28d-b85d3f649964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:41:21.915561 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:21.915526    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:23.034969 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:23.034931    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:25.444989 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:41:25.444954    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:41:26.099494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:26.099446    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:26.281749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:26.281713    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:27.655923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.655886    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"]
Oct 30 03:41:27.656469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.656354    5061 cpu_manager.go:414] "containerMap: removing container" podUID="8428bdda-18a6-4826-8978-b7f888c9e422" containerName="payload-pre-processing"
Oct 30 03:41:27.656469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.656375    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8428bdda-18a6-4826-8978-b7f888c9e422" containerName="payload-pre-processing"
Oct 30 03:41:27.656469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.656424    5061 cpu_manager.go:414] "containerMap: removing container" podUID="afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" containerName="main"
Oct 30 03:41:27.656469 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.656436    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" containerName="main"
Oct 30 03:41:27.656729 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.656478    5061 cpu_manager.go:414] "containerMap: removing container" podUID="829da91a-ed69-47a3-a18f-74d8c27c4c61" containerName="maas-api"
Oct 30 03:41:27.656729 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.656487    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829da91a-ed69-47a3-a18f-74d8c27c4c61" containerName="maas-api"
Oct 30 03:41:27.656729 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.656590    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb90ae7-4344-49d7-ae1b-3d8a8a1540a0" containerName="main"
Oct 30 03:41:27.656729 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.656605    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="829da91a-ed69-47a3-a18f-74d8c27c4c61" containerName="maas-api"
Oct 30 03:41:27.656729 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.656612    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="8428bdda-18a6-4826-8978-b7f888c9e422" containerName="payload-pre-processing"
Oct 30 03:41:27.660337 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.660314    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.676202 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.676178    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-70abcb-serving-cert\""
Oct 30 03:41:27.696145 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.696105    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"]
Oct 30 03:41:27.740274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.740238    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:27.756529 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.756497    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5b2f\" (UniqueName: \"kubernetes.io/projected/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-kube-api-access-l5b2f\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.756721 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.756673    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.756800 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.756718    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.756800 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.756767    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.854394 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.854355    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:27.857671 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.857645    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.857827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.857687    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.857827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.857714    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.857827 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.857763    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5b2f\" (UniqueName: \"kubernetes.io/projected/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-kube-api-access-l5b2f\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.857995 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:41:27.857877    5061 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-serving-cert: secret "maas-api-e2e-worker-single-main-70abcb-serving-cert" not found
Oct 30 03:41:27.857995 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:41:27.857966    5061 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-maas-api-tls podName:4b187bd9-2cb7-45d3-99a4-a18cd8c371d9 nodeName:}" failed. No retries permitted until 2026-10-30 03:41:28.357943863 +0000 UTC m=+3633.660122227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-maas-api-tls") pod "maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g" (UID: "4b187bd9-2cb7-45d3-99a4-a18cd8c371d9") : secret "maas-api-e2e-worker-single-main-70abcb-serving-cert" not found
Oct 30 03:41:27.858385 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.858358    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.860338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.860313    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.868910 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.868884    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"]
Oct 30 03:41:27.872950 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.872932    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:41:27.878379 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.878353    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5b2f\" (UniqueName: \"kubernetes.io/projected/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-kube-api-access-l5b2f\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:27.891256 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.891235    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-70abcb\""
Oct 30 03:41:27.891400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.891276    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b05be57c-dockercfg-xlrd9\""
Oct 30 03:41:27.893616 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.893592    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"]
Oct 30 03:41:27.959234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.959156    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4adc99-6161-476c-977b-775b1d8a3b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz\" (UID: \"7e4adc99-6161-476c-977b-775b1d8a3b36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:41:27.959234 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.959194    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjr4z\" (UniqueName: \"kubernetes.io/projected/7e4adc99-6161-476c-977b-775b1d8a3b36-kube-api-access-pjr4z\") pod \"payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz\" (UID: \"7e4adc99-6161-476c-977b-775b1d8a3b36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:41:27.984964 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:27.984935    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:28.060172 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:28.060136    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4adc99-6161-476c-977b-775b1d8a3b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz\" (UID: \"7e4adc99-6161-476c-977b-775b1d8a3b36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:41:28.060364 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:28.060181    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjr4z\" (UniqueName: \"kubernetes.io/projected/7e4adc99-6161-476c-977b-775b1d8a3b36-kube-api-access-pjr4z\") pod \"payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz\" (UID: \"7e4adc99-6161-476c-977b-775b1d8a3b36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:41:28.061053 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:28.061033    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4adc99-6161-476c-977b-775b1d8a3b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz\" (UID: \"7e4adc99-6161-476c-977b-775b1d8a3b36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:41:28.077831 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:28.077796    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjr4z\" (UniqueName: \"kubernetes.io/projected/7e4adc99-6161-476c-977b-775b1d8a3b36-kube-api-access-pjr4z\") pod \"payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz\" (UID: \"7e4adc99-6161-476c-977b-775b1d8a3b36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:41:28.196330 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:28.196304    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:41:28.363170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:28.363132    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:28.365592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:28.365570    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:28.495960 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:28.495772    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"]
Oct 30 03:41:28.504170 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:41:28.504093    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4adc99_6161_476c_977b_775b1d8a3b36.slice/crio-cfb5d8068cb6f0b58750337f9945ee06b85c87a3933809c4087abbde829cc058 WatchSource:0}: Error finding container cfb5d8068cb6f0b58750337f9945ee06b85c87a3933809c4087abbde829cc058: Status 404 returned error can't find the container with id cfb5d8068cb6f0b58750337f9945ee06b85c87a3933809c4087abbde829cc058
Oct 30 03:41:28.573985 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:28.573944    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:28.887257 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:28.887225    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"]
Oct 30 03:41:28.891647 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:41:28.891615    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b187bd9_2cb7_45d3_99a4_a18cd8c371d9.slice/crio-b10b6385638f1542da6e3be2382beacedae13948bd21b963f8045aaa4fadfbc3 WatchSource:0}: Error finding container b10b6385638f1542da6e3be2382beacedae13948bd21b963f8045aaa4fadfbc3: Status 404 returned error can't find the container with id b10b6385638f1542da6e3be2382beacedae13948bd21b963f8045aaa4fadfbc3
Oct 30 03:41:29.053756 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:29.053710    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g" event={"ID":"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9","Type":"ContainerStarted","Data":"b10b6385638f1542da6e3be2382beacedae13948bd21b963f8045aaa4fadfbc3"}
Oct 30 03:41:29.058286 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:29.058247    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz" event={"ID":"7e4adc99-6161-476c-977b-775b1d8a3b36","Type":"ContainerStarted","Data":"9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d"}
Oct 30 03:41:29.058459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:29.058298    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz" event={"ID":"7e4adc99-6161-476c-977b-775b1d8a3b36","Type":"ContainerStarted","Data":"cfb5d8068cb6f0b58750337f9945ee06b85c87a3933809c4087abbde829cc058"}
Oct 30 03:41:29.058459 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:29.058394    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:41:29.093818 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:29.093762    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz" podStartSLOduration=2.093748829 podStartE2EDuration="2.093748829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:29.092415354 +0000 UTC m=+3634.394593719" watchObservedRunningTime="2026-10-30 03:41:29.093748829 +0000 UTC m=+3634.395927193"
Oct 30 03:41:29.555176 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:29.555145    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:30.065628 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:30.065591    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g" event={"ID":"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9","Type":"ContainerStarted","Data":"94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1"}
Oct 30 03:41:30.066064 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:30.065796    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:30.103498 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:30.103455    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g" podStartSLOduration=2.680957008 podStartE2EDuration="3.103442325s" totalImagesPullingTime="422.485317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:30.1025698 +0000 UTC m=+3635.404748164" watchObservedRunningTime="2026-10-30 03:41:30.103442325 +0000 UTC m=+3635.405620689"
Oct 30 03:41:31.513220 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:31.513179    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:33.149377 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:33.149337    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:34.931313 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:34.931275    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:36.079682 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:36.079605    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:41:36.635568 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:36.635466    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:38.430138 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:38.430032    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:40.068934 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:40.068908    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:41:40.211177 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:40.211143    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:40.885100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:40.885059    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:41.523967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:41.523937    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:41.642350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:41.642316    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:41.784687 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:41.784578    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:42.080766 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.080731    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:42.768756 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.768721    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"]
Oct 30 03:41:42.772564 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.772537    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.786431 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.786409    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-70abcb\"/\"kube-root-ca.crt\""
Oct 30 03:41:42.796761 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.796739    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-70abcb\"/\"openshift-service-ca.crt\""
Oct 30 03:41:42.796919 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.796906    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-70abcb\"/\"simulator-single-main-70abcb-kserve-self-signed-certs\""
Oct 30 03:41:42.797008 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.796919    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-70abcb\"/\"default-dockercfg-7n88z\""
Oct 30 03:41:42.799004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.798984    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"]
Oct 30 03:41:42.818400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.818375    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxnh\" (UniqueName: \"kubernetes.io/projected/43161941-2aa4-4ea1-81ed-297e8c31ecef-kube-api-access-nfxnh\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.818513 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.818422    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-model-cache\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.818558 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.818510    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43161941-2aa4-4ea1-81ed-297e8c31ecef-tls-certs\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.818603 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.818569    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-dshm\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.818691 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.818669    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-tmp-dir\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.818757 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.818732    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-home\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.854592 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.854550    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:42.919710 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.919686    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxnh\" (UniqueName: \"kubernetes.io/projected/43161941-2aa4-4ea1-81ed-297e8c31ecef-kube-api-access-nfxnh\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.919806 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.919722    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-model-cache\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.919806 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.919755    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43161941-2aa4-4ea1-81ed-297e8c31ecef-tls-certs\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.919935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.919810    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-dshm\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.919935 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.919889    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-tmp-dir\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.920036 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.920001    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-home\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.920222 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.920186    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-model-cache\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.920323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.920203    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-tmp-dir\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.920323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.920311    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-home\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.922274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.922250    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-dshm\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.922437 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.922414    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43161941-2aa4-4ea1-81ed-297e8c31ecef-tls-certs\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:42.938967 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:42.938944    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxnh\" (UniqueName: \"kubernetes.io/projected/43161941-2aa4-4ea1-81ed-297e8c31ecef-kube-api-access-nfxnh\") pod \"simulator-single-main-70abcb-kserve-588dbb4f95-7p449\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") " pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:43.086133 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:43.086094    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:43.382160 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:43.382087    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"]
Oct 30 03:41:43.386393 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:41:43.386360    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43161941_2aa4_4ea1_81ed_297e8c31ecef.slice/crio-6f8240f2685a22405baa30362d5478f5bf4a7fe1ac302a0ca99e895b16522844 WatchSource:0}: Error finding container 6f8240f2685a22405baa30362d5478f5bf4a7fe1ac302a0ca99e895b16522844: Status 404 returned error can't find the container with id 6f8240f2685a22405baa30362d5478f5bf4a7fe1ac302a0ca99e895b16522844
Oct 30 03:41:44.032872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:44.032811    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:44.148776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:44.148726    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449" event={"ID":"43161941-2aa4-4ea1-81ed-297e8c31ecef","Type":"ContainerStarted","Data":"f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f"}
Oct 30 03:41:44.148776 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:44.148781    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449" event={"ID":"43161941-2aa4-4ea1-81ed-297e8c31ecef","Type":"ContainerStarted","Data":"6f8240f2685a22405baa30362d5478f5bf4a7fe1ac302a0ca99e895b16522844"}
Oct 30 03:41:44.187831 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:44.187776    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449" podStartSLOduration=2.187762625 podStartE2EDuration="2.187762625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:44.186155627 +0000 UTC m=+3649.488333990" watchObservedRunningTime="2026-10-30 03:41:44.187762625 +0000 UTC m=+3649.489941020"
Oct 30 03:41:45.087136 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:45.087099    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:45.821681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:45.821637    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:45.978790 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:45.978753    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:46.109170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:46.109092    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:47.840462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:47.840419    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:49.717413 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:49.717359    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:51.453325 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:51.453290    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:52.118682 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:52.118649    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:52.238283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:52.238256    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:53.086574 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:53.086539    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:53.099679 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:53.099646    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:53.170120 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:53.170092    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:53.201684 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:53.201660    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:41:54.851012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:54.850977    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:55.444699 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:41:55.444662    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:41:56.496990 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:56.496960    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:58.314706 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:58.314663    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:59.979200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:41:59.979163    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:01.776427 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:01.776393    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:03.569253 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:03.569212    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:05.208216 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:05.208180    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:06.914522 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:06.914486    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:08.531812 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:08.531778    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:10.194335 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:10.194290    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:10.413245 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:10.413210    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:12.039692 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:12.039660    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:13.522992 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:13.522960    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:13.697525 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:13.697471    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:15.633479 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:15.633445    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:17.395916 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:17.395886    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:18.461300 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:18.461265    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"]
Oct 30 03:42:18.553999 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:18.553963    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:19.085805 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:19.085773    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:20.904912 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:20.904876    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:22.525177 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:22.525138    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:23.860697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:23.860659    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"]
Oct 30 03:42:23.861230 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:23.860912    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g" podUID="4b187bd9-2cb7-45d3-99a4-a18cd8c371d9" containerName="maas-api" containerID="cri-o://94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1" gracePeriod=30
Oct 30 03:42:23.994456 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:23.994329    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:24.145070 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:24.144989    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"]
Oct 30 03:42:24.246401 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:24.246354    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:24.378491 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:24.378435    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:25.107285 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.107260    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:42:25.220871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.220776    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-metrics-certs\") pod \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") "
Oct 30 03:42:25.221040 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.220881    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-kube-api-access-l5b2f\" (UniqueName: \"kubernetes.io/projected/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-kube-api-access-l5b2f\") pod \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") "
Oct 30 03:42:25.221040 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.220916    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-openshift-service-ca\") pod \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") "
Oct 30 03:42:25.221040 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.221004    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-maas-api-tls\") pod \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\" (UID: \"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9\") "
Oct 30 03:42:25.221320 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.221294    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-openshift-service-ca" pod "4b187bd9-2cb7-45d3-99a4-a18cd8c371d9" (UID: "4b187bd9-2cb7-45d3-99a4-a18cd8c371d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:25.223011 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.222989    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-maas-api-tls" pod "4b187bd9-2cb7-45d3-99a4-a18cd8c371d9" (UID: "4b187bd9-2cb7-45d3-99a4-a18cd8c371d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:25.223109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.223047    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-metrics-certs" pod "4b187bd9-2cb7-45d3-99a4-a18cd8c371d9" (UID: "4b187bd9-2cb7-45d3-99a4-a18cd8c371d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:25.223109 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.223048    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-kube-api-access-l5b2f" pod "4b187bd9-2cb7-45d3-99a4-a18cd8c371d9" (UID: "4b187bd9-2cb7-45d3-99a4-a18cd8c371d9"). InnerVolumeSpecName "kube-api-access-l5b2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:25.322184 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.322159    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5b2f\" (UniqueName: \"kubernetes.io/projected/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-kube-api-access-l5b2f\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:25.322184 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.322183    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:25.322338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.322193    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:25.322338 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.322202    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:25.337902 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.337872    5061 generic.go:356] "Generic (PLEG): container finished" podID="4b187bd9-2cb7-45d3-99a4-a18cd8c371d9" containerID="94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1" exitCode=0
Oct 30 03:42:25.338013 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.337920    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"
Oct 30 03:42:25.338013 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.337945    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g" event={"ID":"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9","Type":"ContainerDied","Data":"94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1"}
Oct 30 03:42:25.338013 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.338001    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g" event={"ID":"4b187bd9-2cb7-45d3-99a4-a18cd8c371d9","Type":"ContainerDied","Data":"b10b6385638f1542da6e3be2382beacedae13948bd21b963f8045aaa4fadfbc3"}
Oct 30 03:42:25.338125 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.338024    5061 scope.go:122] "RemoveContainer" containerID="94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1"
Oct 30 03:42:25.347783 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.347767    5061 scope.go:122] "RemoveContainer" containerID="94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1"
Oct 30 03:42:25.348070 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:42:25.348049    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1\": container with ID starting with 94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1 not found: ID does not exist" containerID="94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1"
Oct 30 03:42:25.348175 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.348075    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1"} err="failed to get container status \"94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1\": rpc error: code = NotFound desc = could not find container \"94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1\": container with ID starting with 94b006145c9a26e2b681cb815f7f6da4a9988ea97f012f27bffb669d54e6e4c1 not found: ID does not exist"
Oct 30 03:42:25.382955 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.382931    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"]
Oct 30 03:42:25.387082 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.387061    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-70abcb-589767cc96-z4n8g"]
Oct 30 03:42:25.445081 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:42:25.445049    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:42:25.509100 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.509034    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:25.632696 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:25.632652    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:26.159268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:26.159236    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:26.845630 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:26.845596    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b187bd9-2cb7-45d3-99a4-a18cd8c371d9" path="/var/lib/kubelet/pods/4b187bd9-2cb7-45d3-99a4-a18cd8c371d9/volumes"
Oct 30 03:42:29.145744 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:29.145696    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz" podUID="7e4adc99-6161-476c-977b-775b1d8a3b36" containerName="payload-pre-processing" containerID="cri-o://9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d" gracePeriod=25
Oct 30 03:42:30.067480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:30.067439    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz" podUID="7e4adc99-6161-476c-977b-775b1d8a3b36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:42:31.796200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:31.796164    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:31.951274 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:31.951241    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:32.179801 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:32.179716    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:33.481323 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.481282    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449" podUID="43161941-2aa4-4ea1-81ed-297e8c31ecef" containerName="main" containerID="cri-o://f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f" gracePeriod=45
Oct 30 03:42:33.744576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.744524    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:42:33.801370 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.801337    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43161941-2aa4-4ea1-81ed-297e8c31ecef-tls-certs\" (UniqueName: \"kubernetes.io/secret/43161941-2aa4-4ea1-81ed-297e8c31ecef-tls-certs\") pod \"43161941-2aa4-4ea1-81ed-297e8c31ecef\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") "
Oct 30 03:42:33.801538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.801379    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-tmp-dir\") pod \"43161941-2aa4-4ea1-81ed-297e8c31ecef\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") "
Oct 30 03:42:33.801538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.801442    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43161941-2aa4-4ea1-81ed-297e8c31ecef-kube-api-access-nfxnh\" (UniqueName: \"kubernetes.io/projected/43161941-2aa4-4ea1-81ed-297e8c31ecef-kube-api-access-nfxnh\") pod \"43161941-2aa4-4ea1-81ed-297e8c31ecef\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") "
Oct 30 03:42:33.801538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.801475    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-home\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-home\") pod \"43161941-2aa4-4ea1-81ed-297e8c31ecef\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") "
Oct 30 03:42:33.801538 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.801509    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-model-cache\") pod \"43161941-2aa4-4ea1-81ed-297e8c31ecef\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") "
Oct 30 03:42:33.801795 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.801603    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-dshm\") pod \"43161941-2aa4-4ea1-81ed-297e8c31ecef\" (UID: \"43161941-2aa4-4ea1-81ed-297e8c31ecef\") "
Oct 30 03:42:33.801920 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.801669    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-tmp-dir" pod "43161941-2aa4-4ea1-81ed-297e8c31ecef" (UID: "43161941-2aa4-4ea1-81ed-297e8c31ecef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:33.802052 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.801763    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-home" pod "43161941-2aa4-4ea1-81ed-297e8c31ecef" (UID: "43161941-2aa4-4ea1-81ed-297e8c31ecef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:33.802052 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.801888    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-model-cache" pod "43161941-2aa4-4ea1-81ed-297e8c31ecef" (UID: "43161941-2aa4-4ea1-81ed-297e8c31ecef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:33.803606 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.803584    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43161941-2aa4-4ea1-81ed-297e8c31ecef-tls-certs" pod "43161941-2aa4-4ea1-81ed-297e8c31ecef" (UID: "43161941-2aa4-4ea1-81ed-297e8c31ecef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:33.803759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.803699    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-dshm" pod "43161941-2aa4-4ea1-81ed-297e8c31ecef" (UID: "43161941-2aa4-4ea1-81ed-297e8c31ecef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:33.803759 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.803738    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43161941-2aa4-4ea1-81ed-297e8c31ecef-kube-api-access-nfxnh" pod "43161941-2aa4-4ea1-81ed-297e8c31ecef" (UID: "43161941-2aa4-4ea1-81ed-297e8c31ecef"). InnerVolumeSpecName "kube-api-access-nfxnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:33.902885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.902840    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfxnh\" (UniqueName: \"kubernetes.io/projected/43161941-2aa4-4ea1-81ed-297e8c31ecef-kube-api-access-nfxnh\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:33.903050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.902967    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:33.903050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.902990    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:33.903050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.903006    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:33.903050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.903019    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43161941-2aa4-4ea1-81ed-297e8c31ecef-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:33.903050 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:33.903034    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43161941-2aa4-4ea1-81ed-297e8c31ecef-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:34.380896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:34.380864    5061 generic.go:356] "Generic (PLEG): container finished" podID="43161941-2aa4-4ea1-81ed-297e8c31ecef" containerID="f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f" exitCode=0
Oct 30 03:42:34.381079 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:34.380919    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"
Oct 30 03:42:34.381079 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:34.380949    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449" event={"ID":"43161941-2aa4-4ea1-81ed-297e8c31ecef","Type":"ContainerDied","Data":"f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f"}
Oct 30 03:42:34.381079 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:34.380994    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449" event={"ID":"43161941-2aa4-4ea1-81ed-297e8c31ecef","Type":"ContainerDied","Data":"6f8240f2685a22405baa30362d5478f5bf4a7fe1ac302a0ca99e895b16522844"}
Oct 30 03:42:34.381079 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:34.381012    5061 scope.go:122] "RemoveContainer" containerID="f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f"
Oct 30 03:42:34.391381 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:34.391360    5061 scope.go:122] "RemoveContainer" containerID="f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f"
Oct 30 03:42:34.391638 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:42:34.391620    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f\": container with ID starting with f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f not found: ID does not exist" containerID="f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f"
Oct 30 03:42:34.391687 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:34.391647    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f"} err="failed to get container status \"f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f\": rpc error: code = NotFound desc = could not find container \"f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f\": container with ID starting with f0754678bec5c38130d616a7169787572ac2b4c284a286758c8c7bb2e101124f not found: ID does not exist"
Oct 30 03:42:34.434063 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:34.428476    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"]
Oct 30 03:42:34.437463 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:34.437443    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-70abcb/simulator-single-main-70abcb-kserve-588dbb4f95-7p449"]
Oct 30 03:42:34.845805 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:34.845772    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43161941-2aa4-4ea1-81ed-297e8c31ecef" path="/var/lib/kubelet/pods/43161941-2aa4-4ea1-81ed-297e8c31ecef/volumes"
Oct 30 03:42:38.998998 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:38.998898    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"]
Oct 30 03:42:38.999984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:38.999887    5061 cpu_manager.go:414] "containerMap: removing container" podUID="43161941-2aa4-4ea1-81ed-297e8c31ecef" containerName="main"
Oct 30 03:42:38.999984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:38.999916    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43161941-2aa4-4ea1-81ed-297e8c31ecef" containerName="main"
Oct 30 03:42:38.999984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:38.999957    5061 cpu_manager.go:414] "containerMap: removing container" podUID="4b187bd9-2cb7-45d3-99a4-a18cd8c371d9" containerName="maas-api"
Oct 30 03:42:38.999984 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:38.999965    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b187bd9-2cb7-45d3-99a4-a18cd8c371d9" containerName="maas-api"
Oct 30 03:42:39.000177 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.000130    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="43161941-2aa4-4ea1-81ed-297e8c31ecef" containerName="main"
Oct 30 03:42:39.000177 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.000146    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b187bd9-2cb7-45d3-99a4-a18cd8c371d9" containerName="maas-api"
Oct 30 03:42:39.006551 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.006490    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.026732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.023759    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"]
Oct 30 03:42:39.028709 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.028605    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-08208264-kserve-self-signed-certs\""
Oct 30 03:42:39.107229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.107147    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:39.170022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.169907    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tls-certs\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.170569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.170073    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-home\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.170569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.170245    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zggw\" (UniqueName: \"kubernetes.io/projected/e2ee6316-5613-4d38-b20f-ff2b06ed4096-kube-api-access-5zggw\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.170569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.170304    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-model-cache\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.170569 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.170460    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tmp-dir\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.170873 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.170565    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-dshm\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.273393 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.273090    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tls-certs\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.273393 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.273219    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-home\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.273393 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.273297    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zggw\" (UniqueName: \"kubernetes.io/projected/e2ee6316-5613-4d38-b20f-ff2b06ed4096-kube-api-access-5zggw\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.273393 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.273348    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-model-cache\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.274204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.273449    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tmp-dir\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.274204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.273487    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-dshm\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.274204 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.274176    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-home\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.274387 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.274238    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-model-cache\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.274433 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.274381    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tmp-dir\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.277891 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.277829    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-dshm\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.278344 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.278317    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tls-certs\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.294480 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.294415    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zggw\" (UniqueName: \"kubernetes.io/projected/e2ee6316-5613-4d38-b20f-ff2b06ed4096-kube-api-access-5zggw\") pod \"e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") " pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.333590 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.333496    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:39.678179 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:39.678084    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"]
Oct 30 03:42:39.686295 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:42:39.686229    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2ee6316_5613_4d38_b20f_ff2b06ed4096.slice/crio-fab8f3e5d6f2efb066662a92b06c858be93c2eb06969e272f327d4cbafa84fe7 WatchSource:0}: Error finding container fab8f3e5d6f2efb066662a92b06c858be93c2eb06969e272f327d4cbafa84fe7: Status 404 returned error can't find the container with id fab8f3e5d6f2efb066662a92b06c858be93c2eb06969e272f327d4cbafa84fe7
Oct 30 03:42:40.067732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:40.067668    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz" podUID="7e4adc99-6161-476c-977b-775b1d8a3b36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:42:40.422305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:40.422221    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf" event={"ID":"e2ee6316-5613-4d38-b20f-ff2b06ed4096","Type":"ContainerStarted","Data":"3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9"}
Oct 30 03:42:40.422305 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:40.422268    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf" event={"ID":"e2ee6316-5613-4d38-b20f-ff2b06ed4096","Type":"ContainerStarted","Data":"fab8f3e5d6f2efb066662a92b06c858be93c2eb06969e272f327d4cbafa84fe7"}
Oct 30 03:42:40.458455 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:40.458401    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf" podStartSLOduration=2.458388134 podStartE2EDuration="2.458388134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:40.457915407 +0000 UTC m=+3705.760093772" watchObservedRunningTime="2026-10-30 03:42:40.458388134 +0000 UTC m=+3705.760566497"
Oct 30 03:42:41.334041 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:41.334004    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:49.333741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.333708    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:49.350153 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.350125    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:49.398494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.398472    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:42:49.488042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.487968    5061 generic.go:356] "Generic (PLEG): container finished" podID="7e4adc99-6161-476c-977b-775b1d8a3b36" containerID="9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d" exitCode=0
Oct 30 03:42:49.488042 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.488006    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"
Oct 30 03:42:49.488215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.488044    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz" event={"ID":"7e4adc99-6161-476c-977b-775b1d8a3b36","Type":"ContainerDied","Data":"9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d"}
Oct 30 03:42:49.488215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.488104    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz" event={"ID":"7e4adc99-6161-476c-977b-775b1d8a3b36","Type":"ContainerDied","Data":"cfb5d8068cb6f0b58750337f9945ee06b85c87a3933809c4087abbde829cc058"}
Oct 30 03:42:49.488215 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.488172    5061 scope.go:122] "RemoveContainer" containerID="9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d"
Oct 30 03:42:49.497354 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.497335    5061 scope.go:122] "RemoveContainer" containerID="9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d"
Oct 30 03:42:49.497654 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:42:49.497626    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d\": container with ID starting with 9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d not found: ID does not exist" containerID="9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d"
Oct 30 03:42:49.497718 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.497655    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d"} err="failed to get container status \"9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d\": rpc error: code = NotFound desc = could not find container \"9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d\": container with ID starting with 9df4cf77335cfbee9ee57b9986b782553716426208a7f78ebb4bbce0ce4e5c0d not found: ID does not exist"
Oct 30 03:42:49.500583 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.500564    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:42:49.537200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.537178    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4adc99-6161-476c-977b-775b1d8a3b36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4adc99-6161-476c-977b-775b1d8a3b36-plugins-config-volume\") pod \"7e4adc99-6161-476c-977b-775b1d8a3b36\" (UID: \"7e4adc99-6161-476c-977b-775b1d8a3b36\") "
Oct 30 03:42:49.537301 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.537211    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4adc99-6161-476c-977b-775b1d8a3b36-kube-api-access-pjr4z\" (UniqueName: \"kubernetes.io/projected/7e4adc99-6161-476c-977b-775b1d8a3b36-kube-api-access-pjr4z\") pod \"7e4adc99-6161-476c-977b-775b1d8a3b36\" (UID: \"7e4adc99-6161-476c-977b-775b1d8a3b36\") "
Oct 30 03:42:49.537599 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.537577    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4adc99-6161-476c-977b-775b1d8a3b36-plugins-config-volume" pod "7e4adc99-6161-476c-977b-775b1d8a3b36" (UID: "7e4adc99-6161-476c-977b-775b1d8a3b36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:49.539229 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.539206    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4adc99-6161-476c-977b-775b1d8a3b36-kube-api-access-pjr4z" pod "7e4adc99-6161-476c-977b-775b1d8a3b36" (UID: "7e4adc99-6161-476c-977b-775b1d8a3b36"). InnerVolumeSpecName "kube-api-access-pjr4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:49.638987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.638950    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjr4z\" (UniqueName: \"kubernetes.io/projected/7e4adc99-6161-476c-977b-775b1d8a3b36-kube-api-access-pjr4z\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:49.638987 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.638986    5061 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4adc99-6161-476c-977b-775b1d8a3b36-plugins-config-volume\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:42:49.829760 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.829728    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"]
Oct 30 03:42:49.839781 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:49.839755    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-70abcb-649f95ghgz"]
Oct 30 03:42:50.850937 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:50.850901    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4adc99-6161-476c-977b-775b1d8a3b36" path="/var/lib/kubelet/pods/7e4adc99-6161-476c-977b-775b1d8a3b36/volumes"
Oct 30 03:42:52.247773 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:52.247739    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:53.123479 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.123439    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"]
Oct 30 03:42:53.124143 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.124115    5061 cpu_manager.go:414] "containerMap: removing container" podUID="7e4adc99-6161-476c-977b-775b1d8a3b36" containerName="payload-pre-processing"
Oct 30 03:42:53.124272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.124146    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4adc99-6161-476c-977b-775b1d8a3b36" containerName="payload-pre-processing"
Oct 30 03:42:53.124341 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.124302    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4adc99-6161-476c-977b-775b1d8a3b36" containerName="payload-pre-processing"
Oct 30 03:42:53.129409 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.129380    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.144793 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.144770    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"]
Oct 30 03:42:53.153301 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.153267    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-08208264-kserve-self-signed-certs\""
Oct 30 03:42:53.170637 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.170612    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkdw4\" (UniqueName: \"kubernetes.io/projected/f1181463-6f5e-4eeb-bb92-e80712d7120b-kube-api-access-fkdw4\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.170741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.170651    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-model-cache\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.170741 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.170670    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-tmp-dir\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.170818 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.170795    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1181463-6f5e-4eeb-bb92-e80712d7120b-tls-certs\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.170871 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.170826    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-dshm\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.170923 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.170910    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-home\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.206575 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.206549    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:53.272076 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.272046    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1181463-6f5e-4eeb-bb92-e80712d7120b-tls-certs\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.272466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.272082    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-dshm\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.272466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.272132    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-home\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.272466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.272200    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkdw4\" (UniqueName: \"kubernetes.io/projected/f1181463-6f5e-4eeb-bb92-e80712d7120b-kube-api-access-fkdw4\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.272466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.272245    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-model-cache\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.272466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.272269    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-tmp-dir\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.272749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.272615    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-home\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.272749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.272635    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-model-cache\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.272749 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.272669    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-tmp-dir\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.274420 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.274399    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-dshm\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.274624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.274605    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1181463-6f5e-4eeb-bb92-e80712d7120b-tls-certs\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.289717 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.289697    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkdw4\" (UniqueName: \"kubernetes.io/projected/f1181463-6f5e-4eeb-bb92-e80712d7120b-kube-api-access-fkdw4\") pod \"e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") " pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.441110 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.441022    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:53.743681 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:53.743590    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"]
Oct 30 03:42:53.747979 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:42:53.747952    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1181463_6f5e_4eeb_bb92_e80712d7120b.slice/crio-26d7a5d22fa2e81c3cb22e58e25b40e1c4c8dbc276f0c2a1fa6c18d5a48bbc9d WatchSource:0}: Error finding container 26d7a5d22fa2e81c3cb22e58e25b40e1c4c8dbc276f0c2a1fa6c18d5a48bbc9d: Status 404 returned error can't find the container with id 26d7a5d22fa2e81c3cb22e58e25b40e1c4c8dbc276f0c2a1fa6c18d5a48bbc9d
Oct 30 03:42:54.516170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:54.516135    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh" event={"ID":"f1181463-6f5e-4eeb-bb92-e80712d7120b","Type":"ContainerStarted","Data":"dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5"}
Oct 30 03:42:54.516170 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:54.516175    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh" event={"ID":"f1181463-6f5e-4eeb-bb92-e80712d7120b","Type":"ContainerStarted","Data":"26d7a5d22fa2e81c3cb22e58e25b40e1c4c8dbc276f0c2a1fa6c18d5a48bbc9d"}
Oct 30 03:42:54.550632 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:54.550587    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh" podStartSLOduration=1.550574232 podStartE2EDuration="1.550574232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:54.550066976 +0000 UTC m=+3719.852245342" watchObservedRunningTime="2026-10-30 03:42:54.550574232 +0000 UTC m=+3719.852752594"
Oct 30 03:42:55.055578 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:55.055544    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:55.442257 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:55.442167    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:42:55.447384 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:42:55.447353    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:42:56.215414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:42:56.215364    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:01.951038 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:01.950996    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:03.441929 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:03.441897    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:43:03.454814 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:03.454785    5061 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:43:03.597193 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:03.597162    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:43:15.119049 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:15.118958    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:43:15.119049 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:15.119022    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pnfx8" podUID="6f86e91a-cb8a-41cc-8c16-879a462fea69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:43:23.869775 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:23.869737    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"]
Oct 30 03:43:23.925765 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:23.925731    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:25.444916 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:43:25.444879    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:43:26.072867 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:26.068567    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"]
Oct 30 03:43:26.122547 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:26.122510    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:32.128058 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:32.128017    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:43:32.128732 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:32.128692    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-4nnf7" podUID="445e1027-8886-446c-87b5-a8e4ed440a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:43:32.318066 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:32.318017    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:43:32.318282 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:32.318085    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4p7" podUID="f65ce0e1-cada-4a84-a7b7-1da0a01595da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:43:38.890157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:38.890118    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf" podUID="e2ee6316-5613-4d38-b20f-ff2b06ed4096" containerName="main" containerID="cri-o://3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9" gracePeriod=45
Oct 30 03:43:39.140962 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.140901    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:43:39.221012 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.220982    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2ee6316-5613-4d38-b20f-ff2b06ed4096-kube-api-access-5zggw\" (UniqueName: \"kubernetes.io/projected/e2ee6316-5613-4d38-b20f-ff2b06ed4096-kube-api-access-5zggw\") pod \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") "
Oct 30 03:43:39.221167 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.221031    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-model-cache\") pod \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") "
Oct 30 03:43:39.221167 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.221099    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tls-certs\") pod \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") "
Oct 30 03:43:39.221167 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.221119    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-dshm\") pod \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") "
Oct 30 03:43:39.221302 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.221169    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-home\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-home\") pod \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") "
Oct 30 03:43:39.221302 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.221207    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tmp-dir\") pod \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\" (UID: \"e2ee6316-5613-4d38-b20f-ff2b06ed4096\") "
Oct 30 03:43:39.221466 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.221387    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-model-cache" pod "e2ee6316-5613-4d38-b20f-ff2b06ed4096" (UID: "e2ee6316-5613-4d38-b20f-ff2b06ed4096"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:39.221584 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.221463    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-home" pod "e2ee6316-5613-4d38-b20f-ff2b06ed4096" (UID: "e2ee6316-5613-4d38-b20f-ff2b06ed4096"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:39.221654 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.221602    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tmp-dir" pod "e2ee6316-5613-4d38-b20f-ff2b06ed4096" (UID: "e2ee6316-5613-4d38-b20f-ff2b06ed4096"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:39.223373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.223344    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-dshm" pod "e2ee6316-5613-4d38-b20f-ff2b06ed4096" (UID: "e2ee6316-5613-4d38-b20f-ff2b06ed4096"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:39.223373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.223349    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tls-certs" pod "e2ee6316-5613-4d38-b20f-ff2b06ed4096" (UID: "e2ee6316-5613-4d38-b20f-ff2b06ed4096"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:39.223514 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.223379    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ee6316-5613-4d38-b20f-ff2b06ed4096-kube-api-access-5zggw" pod "e2ee6316-5613-4d38-b20f-ff2b06ed4096" (UID: "e2ee6316-5613-4d38-b20f-ff2b06ed4096"). InnerVolumeSpecName "kube-api-access-5zggw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:39.323107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.323078    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.323107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.323104    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.323107 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.323112    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.323303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.323120    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.323303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.323129    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zggw\" (UniqueName: \"kubernetes.io/projected/e2ee6316-5613-4d38-b20f-ff2b06ed4096-kube-api-access-5zggw\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.323303 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.323138    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2ee6316-5613-4d38-b20f-ff2b06ed4096-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:39.850311 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.850273    5061 generic.go:356] "Generic (PLEG): container finished" podID="e2ee6316-5613-4d38-b20f-ff2b06ed4096" containerID="3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9" exitCode=0
Oct 30 03:43:39.850519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.850328    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"
Oct 30 03:43:39.850519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.850340    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf" event={"ID":"e2ee6316-5613-4d38-b20f-ff2b06ed4096","Type":"ContainerDied","Data":"3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9"}
Oct 30 03:43:39.850519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.850373    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf" event={"ID":"e2ee6316-5613-4d38-b20f-ff2b06ed4096","Type":"ContainerDied","Data":"fab8f3e5d6f2efb066662a92b06c858be93c2eb06969e272f327d4cbafa84fe7"}
Oct 30 03:43:39.850519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.850388    5061 scope.go:122] "RemoveContainer" containerID="3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9"
Oct 30 03:43:39.862872 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.862818    5061 scope.go:122] "RemoveContainer" containerID="3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9"
Oct 30 03:43:39.863244 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:43:39.863219    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9\": container with ID starting with 3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9 not found: ID does not exist" containerID="3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9"
Oct 30 03:43:39.863350 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.863254    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9"} err="failed to get container status \"3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9\": rpc error: code = NotFound desc = could not find container \"3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9\": container with ID starting with 3c966fd6df4aad44795400881443e46d8849bbb160a6bdb852b7ac6a025f91e9 not found: ID does not exist"
Oct 30 03:43:39.896334 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.896302    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"]
Oct 30 03:43:39.906738 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:39.906706    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-08208264-kserve-7996d56fb9-6btnf"]
Oct 30 03:43:40.850926 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:40.850824    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2ee6316-5613-4d38-b20f-ff2b06ed4096" path="/var/lib/kubelet/pods/e2ee6316-5613-4d38-b20f-ff2b06ed4096/volumes"
Oct 30 03:43:41.124901 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.124655    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh" podUID="f1181463-6f5e-4eeb-bb92-e80712d7120b" containerName="main" containerID="cri-o://dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5" gracePeriod=45
Oct 30 03:43:41.464164 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.464071    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:43:41.569604 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.569482    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1181463-6f5e-4eeb-bb92-e80712d7120b-kube-api-access-fkdw4\" (UniqueName: \"kubernetes.io/projected/f1181463-6f5e-4eeb-bb92-e80712d7120b-kube-api-access-fkdw4\") pod \"f1181463-6f5e-4eeb-bb92-e80712d7120b\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") "
Oct 30 03:43:41.571476 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.571406    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-dshm\") pod \"f1181463-6f5e-4eeb-bb92-e80712d7120b\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") "
Oct 30 03:43:41.572387 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.572332    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1181463-6f5e-4eeb-bb92-e80712d7120b-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1181463-6f5e-4eeb-bb92-e80712d7120b-tls-certs\") pod \"f1181463-6f5e-4eeb-bb92-e80712d7120b\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") "
Oct 30 03:43:41.572708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.572677    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-model-cache\") pod \"f1181463-6f5e-4eeb-bb92-e80712d7120b\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") "
Oct 30 03:43:41.572956 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.572941    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-home\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-home\") pod \"f1181463-6f5e-4eeb-bb92-e80712d7120b\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") "
Oct 30 03:43:41.573249 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.573222    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-tmp-dir\") pod \"f1181463-6f5e-4eeb-bb92-e80712d7120b\" (UID: \"f1181463-6f5e-4eeb-bb92-e80712d7120b\") "
Oct 30 03:43:41.575666 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.573843    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-tmp-dir" pod "f1181463-6f5e-4eeb-bb92-e80712d7120b" (UID: "f1181463-6f5e-4eeb-bb92-e80712d7120b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:41.577446 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.574944    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-model-cache" pod "f1181463-6f5e-4eeb-bb92-e80712d7120b" (UID: "f1181463-6f5e-4eeb-bb92-e80712d7120b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:41.577579 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.577052    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-home" pod "f1181463-6f5e-4eeb-bb92-e80712d7120b" (UID: "f1181463-6f5e-4eeb-bb92-e80712d7120b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:41.584173 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.584030    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1181463-6f5e-4eeb-bb92-e80712d7120b-kube-api-access-fkdw4" pod "f1181463-6f5e-4eeb-bb92-e80712d7120b" (UID: "f1181463-6f5e-4eeb-bb92-e80712d7120b"). InnerVolumeSpecName "kube-api-access-fkdw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:41.593993 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.593710    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1181463-6f5e-4eeb-bb92-e80712d7120b-tls-certs" pod "f1181463-6f5e-4eeb-bb92-e80712d7120b" (UID: "f1181463-6f5e-4eeb-bb92-e80712d7120b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:41.594677 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.594611    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-dshm" pod "f1181463-6f5e-4eeb-bb92-e80712d7120b" (UID: "f1181463-6f5e-4eeb-bb92-e80712d7120b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:41.712232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.701158    5061 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1181463-6f5e-4eeb-bb92-e80712d7120b-tls-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.712232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.701256    5061 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-model-cache\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.712232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.701275    5061 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-home\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.712232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.701291    5061 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-tmp-dir\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.712232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.701314    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkdw4\" (UniqueName: \"kubernetes.io/projected/f1181463-6f5e-4eeb-bb92-e80712d7120b-kube-api-access-fkdw4\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.712232 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.701336    5061 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1181463-6f5e-4eeb-bb92-e80712d7120b-dshm\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.921925 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.918576    5061 generic.go:356] "Generic (PLEG): container finished" podID="f1181463-6f5e-4eeb-bb92-e80712d7120b" containerID="dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5" exitCode=0
Oct 30 03:43:41.923268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.922679    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh" event={"ID":"f1181463-6f5e-4eeb-bb92-e80712d7120b","Type":"ContainerDied","Data":"dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5"}
Oct 30 03:43:41.923268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.922893    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh" event={"ID":"f1181463-6f5e-4eeb-bb92-e80712d7120b","Type":"ContainerDied","Data":"26d7a5d22fa2e81c3cb22e58e25b40e1c4c8dbc276f0c2a1fa6c18d5a48bbc9d"}
Oct 30 03:43:41.923268 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.922943    5061 scope.go:122] "RemoveContainer" containerID="dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5"
Oct 30 03:43:41.924306 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:41.924069    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"
Oct 30 03:43:42.213906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:42.211957    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"]
Oct 30 03:43:42.213906 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:42.213010    5061 scope.go:122] "RemoveContainer" containerID="dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5"
Oct 30 03:43:42.215085 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:43:42.214311    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5\": container with ID starting with dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5 not found: ID does not exist" containerID="dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5"
Oct 30 03:43:42.215085 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:42.214396    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5"} err="failed to get container status \"dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5\": rpc error: code = NotFound desc = could not find container \"dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5\": container with ID starting with dc0cbfae506b01e0ef07f98a290a4cfd3783ec70dfc3dd8ca67f031966121fe5 not found: ID does not exist"
Oct 30 03:43:42.246904 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:42.245215    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-08208264-kserve-5f79dc845c-k92kh"]
Oct 30 03:43:43.517377 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:43.517287    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1181463-6f5e-4eeb-bb92-e80712d7120b" path="/var/lib/kubelet/pods/f1181463-6f5e-4eeb-bb92-e80712d7120b/volumes"
Oct 30 03:43:46.760796 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:46.760740    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:52.376007 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:43:52.375971    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:55.444881 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:43:55.444832    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:44:02.312157 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:02.312125    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:04.854048 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.854008    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-fflnb"]
Oct 30 03:44:04.854653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.854607    5061 cpu_manager.go:414] "containerMap: removing container" podUID="f1181463-6f5e-4eeb-bb92-e80712d7120b" containerName="main"
Oct 30 03:44:04.854653 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.854628    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1181463-6f5e-4eeb-bb92-e80712d7120b" containerName="main"
Oct 30 03:44:04.854813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.854666    5061 cpu_manager.go:414] "containerMap: removing container" podUID="e2ee6316-5613-4d38-b20f-ff2b06ed4096" containerName="main"
Oct 30 03:44:04.854813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.854675    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ee6316-5613-4d38-b20f-ff2b06ed4096" containerName="main"
Oct 30 03:44:04.854813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.854794    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1181463-6f5e-4eeb-bb92-e80712d7120b" containerName="main"
Oct 30 03:44:04.854813 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.854810    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ee6316-5613-4d38-b20f-ff2b06ed4096" containerName="main"
Oct 30 03:44:04.858271 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.858247    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:04.883241 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.883210    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-fflnb"]
Oct 30 03:44:04.927995 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.927968    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-maas-api-tls\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:04.928148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.928004    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj42k\" (UniqueName: \"kubernetes.io/projected/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-kube-api-access-fj42k\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:04.928148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.928025    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-openshift-service-ca\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:04.928148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:04.928100    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-metrics-certs\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:05.029265 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:05.029235    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-maas-api-tls\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:05.029387 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:05.029272    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj42k\" (UniqueName: \"kubernetes.io/projected/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-kube-api-access-fj42k\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:05.029387 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:05.029290    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-openshift-service-ca\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:05.029387 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:05.029310    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-metrics-certs\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:05.029958 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:05.029927    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-openshift-service-ca\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:05.031715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:05.031691    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-metrics-certs\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:05.031878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:05.031838    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-maas-api-tls\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:05.053774 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:05.053740    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj42k\" (UniqueName: \"kubernetes.io/projected/f3fbba0e-6760-49e4-b310-ce95b2a3f3aa-kube-api-access-fj42k\") pod \"maas-api-595cc648c-fflnb\" (UID: \"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:05.171617 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:05.171559    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:05.467909 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:05.467772    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-fflnb"]
Oct 30 03:44:05.472255 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:44:05.472230    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fbba0e_6760_49e4_b310_ce95b2a3f3aa.slice/crio-66c173c66f9ceb029df5ad8b16d138b4100bf790997e6015953d8c495fb72386 WatchSource:0}: Error finding container 66c173c66f9ceb029df5ad8b16d138b4100bf790997e6015953d8c495fb72386: Status 404 returned error can't find the container with id 66c173c66f9ceb029df5ad8b16d138b4100bf790997e6015953d8c495fb72386
Oct 30 03:44:05.473612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:05.473594    5061 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:44:06.410577 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:06.410538    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb" event={"ID":"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa","Type":"ContainerStarted","Data":"bbfd3eb1cf00254f9a9c97d588e508823048ba4a9dfb776c829b5c518f649c45"}
Oct 30 03:44:06.411085 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:06.410600    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb" event={"ID":"f3fbba0e-6760-49e4-b310-ce95b2a3f3aa","Type":"ContainerStarted","Data":"66c173c66f9ceb029df5ad8b16d138b4100bf790997e6015953d8c495fb72386"}
Oct 30 03:44:06.411085 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:06.410683    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:06.447896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:06.447819    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb" podStartSLOduration=1.9637466369999999 podStartE2EDuration="2.447803249s" totalImagesPullingTime="484.056612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:06.446688758 +0000 UTC m=+3791.748867124" watchObservedRunningTime="2026-10-30 03:44:06.447803249 +0000 UTC m=+3791.749981613"
Oct 30 03:44:12.421341 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:12.421312    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-fflnb"
Oct 30 03:44:19.963448 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:19.963408    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"]
Oct 30 03:44:19.963878 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:19.963675    5061 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6" podUID="0f6d499f-545e-4012-a139-cb0268bef829" containerName="maas-api" containerID="cri-o://b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0" gracePeriod=30
Oct 30 03:44:20.392570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:20.392528    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6" podUID="0f6d499f-545e-4012-a139-cb0268bef829" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.39:8443/health\": dial tcp 10.132.0.39:8443: connect: connection refused"
Oct 30 03:44:21.213272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.213246    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 03:44:21.300676 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.300640    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f6d499f-545e-4012-a139-cb0268bef829-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d499f-545e-4012-a139-cb0268bef829-openshift-service-ca\") pod \"0f6d499f-545e-4012-a139-cb0268bef829\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") "
Oct 30 03:44:21.300881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.300690    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-maas-api-tls\") pod \"0f6d499f-545e-4012-a139-cb0268bef829\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") "
Oct 30 03:44:21.300881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.300736    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-metrics-certs\") pod \"0f6d499f-545e-4012-a139-cb0268bef829\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") "
Oct 30 03:44:21.300881 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.300842    5061 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6d499f-545e-4012-a139-cb0268bef829-kube-api-access-69k8h\" (UniqueName: \"kubernetes.io/projected/0f6d499f-545e-4012-a139-cb0268bef829-kube-api-access-69k8h\") pod \"0f6d499f-545e-4012-a139-cb0268bef829\" (UID: \"0f6d499f-545e-4012-a139-cb0268bef829\") "
Oct 30 03:44:21.301070 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.301030    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6d499f-545e-4012-a139-cb0268bef829-openshift-service-ca" pod "0f6d499f-545e-4012-a139-cb0268bef829" (UID: "0f6d499f-545e-4012-a139-cb0268bef829"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:21.301200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.301182    5061 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d499f-545e-4012-a139-cb0268bef829-openshift-service-ca\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:44:21.302913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.302883    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-metrics-certs" pod "0f6d499f-545e-4012-a139-cb0268bef829" (UID: "0f6d499f-545e-4012-a139-cb0268bef829"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:21.303020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.302939    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-maas-api-tls" pod "0f6d499f-545e-4012-a139-cb0268bef829" (UID: "0f6d499f-545e-4012-a139-cb0268bef829"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:21.303020 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.302971    5061 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6d499f-545e-4012-a139-cb0268bef829-kube-api-access-69k8h" pod "0f6d499f-545e-4012-a139-cb0268bef829" (UID: "0f6d499f-545e-4012-a139-cb0268bef829"). InnerVolumeSpecName "kube-api-access-69k8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:21.401715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.401687    5061 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-maas-api-tls\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:44:21.401715 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.401711    5061 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f6d499f-545e-4012-a139-cb0268bef829-metrics-certs\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:44:21.401896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.401721    5061 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69k8h\" (UniqueName: \"kubernetes.io/projected/0f6d499f-545e-4012-a139-cb0268bef829-kube-api-access-69k8h\") on node \"ip-10-0-136-238.ec2.internal\" DevicePath \"\""
Oct 30 03:44:21.497132 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.497098    5061 generic.go:356] "Generic (PLEG): container finished" podID="0f6d499f-545e-4012-a139-cb0268bef829" containerID="b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0" exitCode=0
Oct 30 03:44:21.497240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.497141    5061 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"
Oct 30 03:44:21.497240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.497178    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6" event={"ID":"0f6d499f-545e-4012-a139-cb0268bef829","Type":"ContainerDied","Data":"b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0"}
Oct 30 03:44:21.497240 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.497221    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6" event={"ID":"0f6d499f-545e-4012-a139-cb0268bef829","Type":"ContainerDied","Data":"0437b72576e15758a123cea70e3bd316ad1fcc45356dccac39886112bf8d4093"}
Oct 30 03:44:21.497371 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.497245    5061 scope.go:122] "RemoveContainer" containerID="b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0"
Oct 30 03:44:21.507379 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.507359    5061 scope.go:122] "RemoveContainer" containerID="b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0"
Oct 30 03:44:21.507645 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:44:21.507627    5061 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0\": container with ID starting with b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0 not found: ID does not exist" containerID="b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0"
Oct 30 03:44:21.507708 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.507654    5061 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0"} err="failed to get container status \"b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0\": rpc error: code = NotFound desc = could not find container \"b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0\": container with ID starting with b3031a35f45b4b9397699555896aea02237664b33696d9ee064be58b7f21beb0 not found: ID does not exist"
Oct 30 03:44:21.547244 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.547219    5061 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"]
Oct 30 03:44:21.551299 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:21.551279    5061 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lpjs6"]
Oct 30 03:44:22.849113 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:22.849073    5061 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6d499f-545e-4012-a139-cb0268bef829" path="/var/lib/kubelet/pods/0f6d499f-545e-4012-a139-cb0268bef829/volumes"
Oct 30 03:44:24.561414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:24.560955    5061 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-54ldl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:44:24.561414 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:24.561020    5061 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-54ldl" podUID="91dfc434-1255-4315-a28d-b85d3f649964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:44:25.444919 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:44:25.444881    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:44:52.547885 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:44:52.547833    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:55.445400 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:44:55.445354    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:45:03.495030 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:03.494991    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dc464c95f-jdm9p_516f14ce-326d-4d8e-a251-6f1466249209/authorino/0.log"
Oct 30 03:45:15.937336 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:15.937305    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-5hfrf_28c8ee4f-5f39-4539-abc8-efc5679db424/copy-manifests/0.log"
Oct 30 03:45:15.973517 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:15.973492    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-5hfrf_28c8ee4f-5f39-4539-abc8-efc5679db424/manager/0.log"
Oct 30 03:45:17.475181 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:17.475154    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-fflnb_f3fbba0e-6760-49e4-b310-ce95b2a3f3aa/maas-api/0.log"
Oct 30 03:45:21.459833 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:21.459798    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dc464c95f-jdm9p_516f14ce-326d-4d8e-a251-6f1466249209/authorino/0.log"
Oct 30 03:45:22.115647 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:22.115616    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vhx97_b1608971-f5a4-43b3-8a16-e35dd3bc42be/kuadrant-console-plugin/0.log"
Oct 30 03:45:24.242137 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:24.242105    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64f884f994-jv42m_9eda807c-afcf-48fb-adee-e8881850f7a8/kube-auth-proxy/0.log"
Oct 30 03:45:24.436203 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:24.436174    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-4nnf7_445e1027-8886-446c-87b5-a8e4ed440a06/istio-proxy/0.log"
Oct 30 03:45:24.661511 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:24.661482    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-54ldl_91dfc434-1255-4315-a28d-b85d3f649964/istio-proxy/0.log"
Oct 30 03:45:25.318122 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:25.318089    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-cj4p7_f65ce0e1-cada-4a84-a7b7-1da0a01595da/istio-proxy/0.log"
Oct 30 03:45:25.445587 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:45:25.445523    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:45:25.514612 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:25.514586    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-pnfx8_6f86e91a-cb8a-41cc-8c16-879a462fea69/istio-proxy/0.log"
Oct 30 03:45:26.721311 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:26.721280    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-v8fgz_87148799-07e3-41d4-a6f0-0e4678d90202/payload-processing/0.log"
Oct 30 03:45:26.925402 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:26.925373    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b959b9795-jw4rz_6c28bc38-2c97-412f-89b0-f7a552b78373/router/0.log"
Oct 30 03:45:28.303243 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:28.303214    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-67pfh_e686dfeb-afd4-4de3-8528-60bb29527f1e/main/0.log"
Oct 30 03:45:28.672045 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:28.671971    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq_47b66266-1f25-4615-984e-ebd9fbfb5cac/main/0.log"
Oct 30 03:45:28.861613 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:28.861586    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf_24ea5b2c-abf7-4857-86c5-8c1690ee76f0/main/0.log"
Oct 30 03:45:29.727862 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:29.727821    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dc464c95f-jdm9p_516f14ce-326d-4d8e-a251-6f1466249209/authorino/0.log"
Oct 30 03:45:40.547725 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:40.547692    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-5hfrf_28c8ee4f-5f39-4539-abc8-efc5679db424/copy-manifests/0.log"
Oct 30 03:45:40.584700 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:40.584673    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75bcb9cf-5hfrf_28c8ee4f-5f39-4539-abc8-efc5679db424/manager/0.log"
Oct 30 03:45:41.848022 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:41.847978    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-fflnb_f3fbba0e-6760-49e4-b310-ce95b2a3f3aa/maas-api/0.log"
Oct 30 03:45:45.060056 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:45.060023    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dc464c95f-jdm9p_516f14ce-326d-4d8e-a251-6f1466249209/authorino/0.log"
Oct 30 03:45:45.678913 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:45.678884    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vhx97_b1608971-f5a4-43b3-8a16-e35dd3bc42be/kuadrant-console-plugin/0.log"
Oct 30 03:45:47.465870 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:47.465820    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64f884f994-jv42m_9eda807c-afcf-48fb-adee-e8881850f7a8/kube-auth-proxy/0.log"
Oct 30 03:45:47.655991 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:47.655958    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-4nnf7_445e1027-8886-446c-87b5-a8e4ed440a06/istio-proxy/0.log"
Oct 30 03:45:47.847400 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:47.847367    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-54ldl_91dfc434-1255-4315-a28d-b85d3f649964/istio-proxy/0.log"
Oct 30 03:45:48.433778 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:48.433751    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-cj4p7_f65ce0e1-cada-4a84-a7b7-1da0a01595da/istio-proxy/0.log"
Oct 30 03:45:48.628318 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:48.628291    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-pnfx8_6f86e91a-cb8a-41cc-8c16-879a462fea69/istio-proxy/0.log"
Oct 30 03:45:49.799672 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:49.799619    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-v8fgz_87148799-07e3-41d4-a6f0-0e4678d90202/payload-processing/0.log"
Oct 30 03:45:50.008246 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:50.008214    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b959b9795-jw4rz_6c28bc38-2c97-412f-89b0-f7a552b78373/router/0.log"
Oct 30 03:45:51.095283 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:51.095253    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-67pfh_e686dfeb-afd4-4de3-8528-60bb29527f1e/main/0.log"
Oct 30 03:45:51.279579 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:51.279550    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:51.517374 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:51.517292    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-c2bkq_47b66266-1f25-4615-984e-ebd9fbfb5cac/main/0.log"
Oct 30 03:45:51.702877 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:51.702835    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qk8cf_24ea5b2c-abf7-4857-86c5-8c1690ee76f0/main/0.log"
Oct 30 03:45:52.586001 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:52.585968    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:55.465088 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:45:55.464988    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:45:55.567368 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:55.567284    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:45:55.571932 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:55.571871    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:45:55.572421 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:55.572349    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:45:55.577186 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:45:55.577137    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpwcs_816aaac5-8df6-402e-9367-a2ea2f8ca686/ovn-acl-logging/0.log"
Oct 30 03:46:22.681517 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:46:22.681483    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-v8fgz_87148799-07e3-41d4-a6f0-0e4678d90202/payload-processing/0.log"
Oct 30 03:46:22.868697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:46:22.868671    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-v8fgz_87148799-07e3-41d4-a6f0-0e4678d90202/payload-processing/0.log"
Oct 30 03:46:25.445170 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:46:25.445137    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:46:52.736043 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:46:52.736004    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:55.444750 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:46:55.444715    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:47:03.198952 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.198918    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-v8fgz_87148799-07e3-41d4-a6f0-0e4678d90202/payload-processing/0.log"
Oct 30 03:47:03.375118 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.375090    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-v8fgz_87148799-07e3-41d4-a6f0-0e4678d90202/payload-processing/0.log"
Oct 30 03:47:03.709004 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.708973    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zst6n/must-gather-fbpgg"]
Oct 30 03:47:03.709462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.709442    5061 cpu_manager.go:414] "containerMap: removing container" podUID="0f6d499f-545e-4012-a139-cb0268bef829" containerName="maas-api"
Oct 30 03:47:03.709462 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.709461    5061 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6d499f-545e-4012-a139-cb0268bef829" containerName="maas-api"
Oct 30 03:47:03.709705 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.709560    5061 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6d499f-545e-4012-a139-cb0268bef829" containerName="maas-api"
Oct 30 03:47:03.712570 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.712549    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:03.712828 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.712813    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zst6n/must-gather-fbpgg"
Oct 30 03:47:03.731356 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.731335    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zst6n\"/\"kube-root-ca.crt\""
Oct 30 03:47:03.741237 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.741215    5061 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zst6n\"/\"default-dockercfg-jx5vb\""
Oct 30 03:47:03.741341 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.741244    5061 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zst6n\"/\"openshift-service-ca.crt\""
Oct 30 03:47:03.754519 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.754498    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zst6n/must-gather-fbpgg"]
Oct 30 03:47:03.816542 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.816518    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ceead7a-3605-40a9-bb93-45f34b5727fe-must-gather-output\") pod \"must-gather-fbpgg\" (UID: \"2ceead7a-3605-40a9-bb93-45f34b5727fe\") " pod="openshift-must-gather-zst6n/must-gather-fbpgg"
Oct 30 03:47:03.816699 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.816559    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsm9x\" (UniqueName: \"kubernetes.io/projected/2ceead7a-3605-40a9-bb93-45f34b5727fe-kube-api-access-lsm9x\") pod \"must-gather-fbpgg\" (UID: \"2ceead7a-3605-40a9-bb93-45f34b5727fe\") " pod="openshift-must-gather-zst6n/must-gather-fbpgg"
Oct 30 03:47:03.873615 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.873581    5061 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:03.917272 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.917249    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ceead7a-3605-40a9-bb93-45f34b5727fe-must-gather-output\") pod \"must-gather-fbpgg\" (UID: \"2ceead7a-3605-40a9-bb93-45f34b5727fe\") " pod="openshift-must-gather-zst6n/must-gather-fbpgg"
Oct 30 03:47:03.917373 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.917287    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsm9x\" (UniqueName: \"kubernetes.io/projected/2ceead7a-3605-40a9-bb93-45f34b5727fe-kube-api-access-lsm9x\") pod \"must-gather-fbpgg\" (UID: \"2ceead7a-3605-40a9-bb93-45f34b5727fe\") " pod="openshift-must-gather-zst6n/must-gather-fbpgg"
Oct 30 03:47:03.917571 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.917554    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ceead7a-3605-40a9-bb93-45f34b5727fe-must-gather-output\") pod \"must-gather-fbpgg\" (UID: \"2ceead7a-3605-40a9-bb93-45f34b5727fe\") " pod="openshift-must-gather-zst6n/must-gather-fbpgg"
Oct 30 03:47:03.937097 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:03.937077    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsm9x\" (UniqueName: \"kubernetes.io/projected/2ceead7a-3605-40a9-bb93-45f34b5727fe-kube-api-access-lsm9x\") pod \"must-gather-fbpgg\" (UID: \"2ceead7a-3605-40a9-bb93-45f34b5727fe\") " pod="openshift-must-gather-zst6n/must-gather-fbpgg"
Oct 30 03:47:04.026788 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:04.026678    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zst6n/must-gather-fbpgg"
Oct 30 03:47:04.322835 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:04.322796    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zst6n/must-gather-fbpgg"]
Oct 30 03:47:04.327453 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:47:04.327428    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ceead7a_3605_40a9_bb93_45f34b5727fe.slice/crio-75ef558c364614c689d92dd4e2bfea76b31e47816df992a4cfe818eafac035d2 WatchSource:0}: Error finding container 75ef558c364614c689d92dd4e2bfea76b31e47816df992a4cfe818eafac035d2: Status 404 returned error can't find the container with id 75ef558c364614c689d92dd4e2bfea76b31e47816df992a4cfe818eafac035d2
Oct 30 03:47:04.338436 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:04.338407    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zst6n/must-gather-fbpgg" event={"ID":"2ceead7a-3605-40a9-bb93-45f34b5727fe","Type":"ContainerStarted","Data":"75ef558c364614c689d92dd4e2bfea76b31e47816df992a4cfe818eafac035d2"}
Oct 30 03:47:05.344176 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:05.344144    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zst6n/must-gather-fbpgg" event={"ID":"2ceead7a-3605-40a9-bb93-45f34b5727fe","Type":"ContainerStarted","Data":"bc48499f3ff44fbe00a10712ca00de30dd207911208ae8af5b8e8e4ca4c21f5a"}
Oct 30 03:47:06.350224 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:06.350190    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zst6n/must-gather-fbpgg" event={"ID":"2ceead7a-3605-40a9-bb93-45f34b5727fe","Type":"ContainerStarted","Data":"3a4a67421b7131ef307cd56b518284a5968d20d8ba13dc5924d385ffd7d45927"}
Oct 30 03:47:06.383697 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:06.383650    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zst6n/must-gather-fbpgg" podStartSLOduration=2.554088551 podStartE2EDuration="3.383635799s" totalImagesPullingTime="829.547248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:06.383466609 +0000 UTC m=+3971.685644975" watchObservedRunningTime="2026-10-30 03:47:06.383635799 +0000 UTC m=+3971.685814164"
Oct 30 03:47:10.629260 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:10.629182    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hzqcm_a2bf413f-45b3-450c-b797-4e674e8da8ec/global-pull-secret-syncer/0.log"
Oct 30 03:47:10.958403 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:10.958297    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7btgh_4844aa91-37e4-4cae-a380-af0000c768fc/connection-checker/0.log"
Oct 30 03:47:11.074481 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:11.074450    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cvtzh_bf8a2708-a7fe-41b0-8f63-7850df383c8f/connection-checker/0.log"
Oct 30 03:47:11.193369 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:11.193340    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-j86rx_1ea0be63-7705-40e8-b2a1-e48c39259c8b/connection-checker/0.log"
Oct 30 03:47:11.336624 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:11.336573    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5l2pw_caeb5235-93eb-4b90-8372-1c59c807485f/konnectivity-agent/0.log"
Oct 30 03:47:11.928659 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:11.928595    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-238.ec2.internal_4f18f0dafccedcaa5cd8761c40142410/haproxy/0.log"
Oct 30 03:47:18.628869 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:18.627804    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dc464c95f-jdm9p_516f14ce-326d-4d8e-a251-6f1466249209/authorino/0.log"
Oct 30 03:47:18.982650 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:18.982570    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vhx97_b1608971-f5a4-43b3-8a16-e35dd3bc42be/kuadrant-console-plugin/0.log"
Oct 30 03:47:21.005148 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.005109    5061 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"]
Oct 30 03:47:21.011933 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.011903    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.028896 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.028831    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"]
Oct 30 03:47:21.057003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.056639    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-sys\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.057003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.056726    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-lib-modules\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.057003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.056757    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbmxh\" (UniqueName: \"kubernetes.io/projected/8cd545f4-acc4-46f0-8625-7425006dbc43-kube-api-access-gbmxh\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.057003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.056810    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-podres\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.057003 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.056884    5061 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-proc\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.158035 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.157999    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-lib-modules\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.158223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.158047    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbmxh\" (UniqueName: \"kubernetes.io/projected/8cd545f4-acc4-46f0-8625-7425006dbc43-kube-api-access-gbmxh\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.158223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.158097    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-podres\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.158223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.158131    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-proc\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.158223 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.158151    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-lib-modules\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.158418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.158224    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-podres\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.158418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.158240    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-proc\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.158418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.158273    5061 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-sys\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.158418 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.158331    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd545f4-acc4-46f0-8625-7425006dbc43-sys\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.176156 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.176122    5061 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbmxh\" (UniqueName: \"kubernetes.io/projected/8cd545f4-acc4-46f0-8625-7425006dbc43-kube-api-access-gbmxh\") pod \"perf-node-gather-daemonset-5gmz2\" (UID: \"8cd545f4-acc4-46f0-8625-7425006dbc43\") " pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.330576 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.330539    5061 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:21.635583 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.635502    5061 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"]
Oct 30 03:47:21.639505 ip-10-0-136-238 kubenswrapper[5061]: W1030 03:47:21.639467    5061 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8cd545f4_acc4_46f0_8625_7425006dbc43.slice/crio-00f5fe29c4fea1538d9fee86c318ca54fe05a2a71d7a05f5fc86f13e7f253e91 WatchSource:0}: Error finding container 00f5fe29c4fea1538d9fee86c318ca54fe05a2a71d7a05f5fc86f13e7f253e91: Status 404 returned error can't find the container with id 00f5fe29c4fea1538d9fee86c318ca54fe05a2a71d7a05f5fc86f13e7f253e91
Oct 30 03:47:21.661786 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:21.661764    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-gl8sd_c9aa32ac-0748-4d2f-820d-8a15ec78cdb1/cluster-monitoring-operator/0.log"
Oct 30 03:47:22.524598 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:22.524556    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2" event={"ID":"8cd545f4-acc4-46f0-8625-7425006dbc43","Type":"ContainerStarted","Data":"428ae78f880f6ae9419633a73db4139509e6e213deb8c4b1c48b310cb1608486"}
Oct 30 03:47:22.525043 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:22.524638    5061 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2" event={"ID":"8cd545f4-acc4-46f0-8625-7425006dbc43","Type":"ContainerStarted","Data":"00f5fe29c4fea1538d9fee86c318ca54fe05a2a71d7a05f5fc86f13e7f253e91"}
Oct 30 03:47:22.525043 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:22.524659    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:22.559200 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:22.559147    5061 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2" podStartSLOduration=2.559132711 podStartE2EDuration="2.559132711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:22.557757968 +0000 UTC m=+3987.859936331" watchObservedRunningTime="2026-10-30 03:47:22.559132711 +0000 UTC m=+3987.861311075"
Oct 30 03:47:23.014299 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:23.014262    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b4fjt_f3d6304c-30e6-4f4f-9327-3c4db4beac53/node-exporter/0.log"
Oct 30 03:47:23.126258 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:23.126232    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b4fjt_f3d6304c-30e6-4f4f-9327-3c4db4beac53/kube-rbac-proxy/0.log"
Oct 30 03:47:23.238784 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:23.238755    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b4fjt_f3d6304c-30e6-4f4f-9327-3c4db4beac53/init-textfile/0.log"
Oct 30 03:47:25.447144 ip-10-0-136-238 kubenswrapper[5061]: E1030 03:47:25.447104    5061 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkfq8" containerName="extractor"
Oct 30 03:47:27.121810 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:27.121780    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-sj78j_69caed51-464d-4dab-b4a4-b4ff77da5e4e/networking-console-plugin/0.log"
Oct 30 03:47:28.145117 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:28.145088    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/1.log"
Oct 30 03:47:28.149968 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:28.149945    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-888jp_7b6386c9-d170-4b89-bb41-9a0641207188/console-operator/2.log"
Oct 30 03:47:28.540246 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:28.540176    5061 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zst6n/perf-node-gather-daemonset-5gmz2"
Oct 30 03:47:30.276287 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:30.276261    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-cd7dw_2967919e-d201-48ae-b388-85fe023e1d21/volume-data-source-validator/0.log"
Oct 30 03:47:33.009577 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:33.009008    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m9rxl_16395f11-a919-4dd3-9d21-3388d472e616/dns/0.log"
Oct 30 03:47:33.120720 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:33.120687    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m9rxl_16395f11-a919-4dd3-9d21-3388d472e616/kube-rbac-proxy/0.log"
Oct 30 03:47:33.471118 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:33.471085    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qvbmn_5d7330ef-3e60-407f-89ca-0c0c9e38c87d/dns-node-resolver/0.log"
Oct 30 03:47:34.858326 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:34.858301    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hqwcj_af778722-0e75-4ddd-9819-d783cfa858e1/node-ca/0.log"
Oct 30 03:47:35.717073 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:35.717046    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64f884f994-jv42m_9eda807c-afcf-48fb-adee-e8881850f7a8/kube-auth-proxy/0.log"
Oct 30 03:47:35.778031 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:35.777998    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-4nnf7_445e1027-8886-446c-87b5-a8e4ed440a06/istio-proxy/0.log"
Oct 30 03:47:35.837216 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:35.837189    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-54ldl_91dfc434-1255-4315-a28d-b85d3f649964/istio-proxy/0.log"
Oct 30 03:47:36.027494 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:36.027413    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-cj4p7_f65ce0e1-cada-4a84-a7b7-1da0a01595da/istio-proxy/0.log"
Oct 30 03:47:36.084979 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:36.084952    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-pnfx8_6f86e91a-cb8a-41cc-8c16-879a462fea69/istio-proxy/0.log"
Oct 30 03:47:36.509310 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:36.509280    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-v8fgz_87148799-07e3-41d4-a6f0-0e4678d90202/payload-processing/0.log"
Oct 30 03:47:36.576826 ip-10-0-136-238 kubenswrapper[5061]: I1030 03:47:36.576794    5061 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b959b9795-jw4rz_6c28bc38-2c97-412f-89b0-f7a552b78373/router/0.log"